# Homelab Kubernetes Pipeline This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to Argo CD. ## Architecture The lab is intentionally small but production-shaped: - a Debian amd64 host runs the kubeadm control plane and local deployment tools - a Raspberry Pi arm64 node runs selected workloads - the Debian host also runs the always-on Gitea Docker service outside Kubernetes - the Debian host keeps a bare GitOps mirror under `/home/jv/git-server/my-homelab-configs.git` - a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker templates - OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge - Argo CD continuously reconciles Kubernetes manifests from this repo - a local registry stores the website and demos images built for the worker architecture - the website image keeps Apache as the public server, uses PHP-FPM with OPcache for dynamic endpoints, and pre-renders read-heavy pages to static HTML during the image build - SOPS with age is the committed secret-management path for future encrypted Kubernetes secrets - local Jeannie/RAG eval cases exercise troubleshooting explanations and retrieval grounding without touching live infrastructure - an OCI jump box provides the public edge path back into the homelab over Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream keepalive ## Inventory The canonical non-secret inventory is [homelab.yml](homelab.yml). Keep LAN IPs, Tailscale IPs, public hostnames, ports, storage paths, and service ownership there before copying values into scripts, OpenTofu variables, runbooks, or service docs. Secrets and tokens do not belong in that file. `./{{ main_script }}` reads `homelab.yml` at startup and exports matching OpenTofu variables for the edge, provisioning, cluster, platform, and apps stacks. This keeps values such as the Debian LAN IP, Debian Tailscale IP, RPi LAN IP, OCI IP, Traefik MetalLB IP, and domain name from drifting across scripts and templates. `scripts/render-docs` also renders [docs/service-catalog.md](docs/service-catalog.md) and the static service catalog page at `apps/demos-static/public/homelab-catalog/index.html` from the same inventory. Run `./{{ main_script }} up` and `./{{ main_script }} nuke` only from the Debian homelab server. The script intentionally refuses to run from non-Debian machines so a laptop cannot accidentally modify the cluster. ## Flow 1. `bootstrap/provisioning` - prepares a Debian server as a PXE and preseed service for arm64 VMs - serves Debian 13 arm64 netboot assets through TFTP and HTTP - exposes a PXE rescue menu and HTTP rescue page for manual repair paths - creates a golden image install path with Kubernetes, containerd, qemu-guest-agent, cloud-init, and storage client packages ready - is driven by `./{{ main_script }} up` when Pimox is reachable, without changing Orange Pi host networking 2. `bootstrap/cluster` - creates the kubeadm control plane on the Debian amd64 node - joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes - configures Calico-compatible pod CIDR - configures containerd to pull from the in-cluster NodePort registry - creates retained host directories under `/data/openebs/local` 3. `bootstrap/platform` - installs a minimal Calico deployment through the Tigera operator - installs NodeLocal DNSCache for node-local DNS query caching - installs MetalLB for LAN `LoadBalancer` services - installs Traefik as the single Kubernetes ingress entry point - installs OpenEBS - creates `openebs-hostpath-retain` - installs Argo CD - installs Kyverno with audit-first baseline Pod Security policies and homelab image signature/SBOM admission controls - registers the private GitOps repo without storing the SSH private key in Terraform state 4. `bootstrap/apps` - registers Argo CD Applications from the `applications` map - passes the website image produced by the build step into Argo CD as a Kustomize image override - deploys the tuned website runtime: Apache listens on port 8080, PHP-FPM listens on loopback inside the container, OPcache is enabled, and pre-rendered HTML handles the read-heavy pages - default apps are `container-registry`, `website-production`, `demos-static`, the `heimdall` route bundle, `n8n`, and `supply-chain-policy` 5. `bootstrap/edge` - connects to the OCI jump box - uploads nginx, HAProxy, Varnish, and Squid configs - obtains and renews Let's Encrypt certificates for the configured hostname - runs the edge cache/proxy chain with Docker Compose - enables gzip, HTTP/2, TLS session reuse, client keepalive, upstream keepalive, and cache headers for the public website path ## Prerequisites On the Debian host: - OpenTofu - Docker with Buildx - kubeadm, kubelet, kubectl, and containerd pinned to the target Kubernetes minor, currently `v1.36` - SSH access to worker nodes - SSH access to the OCI edge host - Go toolchain for Jeannie core helpers; `./{{ main_script }} up` installs `golang-go` on the Debian host if it is missing - enough persistent storage for `/data/openebs/local` and Docker's data root After a clean Debian reinstall, bootstrap the host package set and desktop configuration first: ```bash sudo apt-get update sudo apt-get install -y git ansible sudo git clone /home/jv/my-homelab-configs cd /home/jv/my-homelab-configs ansible-playbook -K -i bootstrap/host/inventory.ini bootstrap/host/playbook.yml ``` If you kept the private app-config archive generated from the old Debian host, pass its generated `ansible-vars.yml` with `-e @/path/to/ansible-vars.yml`. Details are in `bootstrap/host/README.md`. The default kubeconfig path is `/home/jv/.kube/config`. Override it with `KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed. ## Implementation Language Policy Jeannie keeps `./{{ main_script }}` as the stable command entrypoint. New durable read-only orchestration, report rendering, inventory parsing, status/doctor logic, and self-heal planning should prefer Go under `cmd/jeannie-core` because it ships as one small binary and is easier to test than Bash. Bash remains for thin compatibility wrappers and remote shell glue. Python remains acceptable for AI/RAG, data-heavy utilities, or cases where an existing Python ecosystem is the clear fit. New repo features should choose the smallest language that fits the task instead of defaulting to Bash. ## Version Discipline The lab should converge on one Kubernetes minor across the API server and every kubelet. The Pimox golden-image default is `v1.36` so rebuilt arm64 VM workers match the Debian control plane and Raspberry Pi worker instead of staying on an older template. Avoid treating a three-minor kubelet skew as normal steady state: it is only a temporary compatibility window and blocks the next control plane upgrade. Check node versions from the Debian host: ```bash ./{{ main_script }} doctor-versions ``` The check prints the API server, kubelet versions, kubelet minors, and containerd runtimes. It exits nonzero when kubelet minors differ from the API server minor or when containerd major versions are mixed. The Gitea Actions auto path runs this check before app deploys; kubelet minor drift switches the job to `./{{ main_script }} rebuild-cluster` so the Pimox template and worker VMs are replaced from the pinned image. For Pimox workers, prefer rebuilding from the corrected golden template and rejoining the worker over ad hoc live package upgrades. ## AI Evals The repo includes a local eval harness for Jeannie explanations and homelab RAG grounding. It uses fixtures and allowlisted read-only commands, so it is safe to run before changing the lab: ```bash ./{{ main_script }} ai-evals run ``` Use `./{{ main_script }} ai-evals list` to see the current cases. Add new cases under `infra/ai-evals` whenever a troubleshooting pattern or runbook behavior should become repeatable. ## Prompt Injection Lab The defensive security training path includes local prompt-injection fixtures for hostile runbooks, tool output, dashboard annotations, and command arguments. Run it without touching live infrastructure: ```bash ./{{ main_script }} prompt-injection-lab run ``` Cases live under `security/prompt-injection-lab` and are meant to become regression tests for future Jeannie AI features. ## Incident Commander The repo also includes a read-only incident commander trainer. It classifies pasted failure text, prints a likely hypothesis, links the right runbook, and lists the next read-only Jeannie commands: ```bash ./{{ main_script }} incident replay ./{{ main_script }} incident triage --text "502 Bad Gateway from nginx" ``` Incident fixtures live under `infra/incident-commander`. Use them to turn repeat outages into repeatable triage behavior before adding any self-healing. ## Safety Cases Before high-blast-radius changes, generate a safety case from the Git diff: ```bash ./{{ main_script }} safety-case --since HEAD~1 ``` It summarizes risk, blast radius, required validation, rollback, and forbidden actions. The implementation lives under `infra/safety-cases` and reuses the same impact rules as `./{{ main_script }} impact`. ## Agent Sandbox Future Jeannie AI actions should be checked before execution. The local sandbox policy starts with a read-only command classifier: ```bash ./{{ main_script }} agent-sandbox check "./{{ main_script }} status" ./{{ main_script }} agent-sandbox check "./{{ main_script }} nuke" ``` Policy is stored in `infra/agent-sandbox/policy.tsv`. ## AI Workload Scheduling For AI-ish workloads, Jeannie can simulate placement across Debian, Pimox workers, and the RPi based on CPU, memory, architecture, tailnet access, and control-plane policy: ```bash ./{{ main_script }} ai-scheduler recommend small-rag ./{{ main_script }} ai-scheduler simulate ``` The simulator lives under `infra/ai-scheduler` and is a stepping stone toward a real Kubernetes controller. ## Memory With Provenance Jeannie can return cited retrieval context and check whether the local RAG index is stale: ```bash ./{{ main_script }} ask --citations "how do I recover the cluster?" ./{{ main_script }} ai-memory check ``` This keeps local AI answers grounded in repo files and runbooks. ## Red/Blue Loop Safe red-team/blue-team checks are grouped under one command: ```bash ./{{ main_script }} red-blue plan ./{{ main_script }} red-blue run --local ``` Local mode runs deterministic prompt-injection and eval checks. Live scans are listed but gated so they can run from the Debian host when intended. ## Canary Promotion The promotion helper defines release gates before production changes: ```bash ./{{ main_script }} promote plan ./{{ main_script }} promote validate ./{{ main_script }} promote rollback website-production ``` The current version is intentionally conservative: it validates local gates and prints rollback steps while dev namespace automation remains explicit future work. ## Guarded Self-Healing Jeannie can turn status findings into an experimental self-heal plan: ```bash ./{{ main_script }} status --heal-plan ./{{ main_script }} status --heal ./{{ main_script }} heal apply --yes ``` Normal `status` remains read-only. The heal planner maps known findings to guarded actions, chooses one highest-impact auto-eligible finding, checks the proposed command against `infra/agent-sandbox/policy.tsv`, and runs only that one action. Re-run `./{{ main_script }} status` after every heal cycle so Jeannie can reassess before touching the next issue. If a specific Pimox worker is NotReady, heal targets that worker with `./{{ main_script }} workers restart N` instead of repeating a broad cluster start. After an action runs, Jeannie waits for the target condition to clear and reports `healed` or `still failing`. Failed heal actions are placed on a short cooldown so Jeannie does not loop on the same VM restart; the next plan escalates to diagnostics instead. Destructive actions remain blocked. ## Model Behavior Observatory Stable prompts track local model behavior, latency, refusal behavior, and tool-correctness signals over time: ```bash ./{{ main_script }} model-observe prompts ./{{ main_script }} model-observe run --offline ./{{ main_script }} model-observe report ``` Use `--live` only when Ollama is available. ## Deploying From the Debian server: ```bash cd ~/my-homelab-configs ./{{ main_script }} up ``` The script first deploys external Gitea to the Debian host with Docker Compose under `/data/homelab-gitea`, backed by the HP laptop NVMe, so Git stays outside the Kubernetes rebuild blast radius. It then detects the Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13 arm64 template, creates or reuses one worker VM clone, discovers the guest IP through qemu-guest-agent, and passes that worker into the cluster layer. It then applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the local registry, builds the website and demos images when their source changed, pushes them to the registry, recreates pods only after a new image is built, and applies the edge stack. Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set `LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline keeps the template on its configured `local` storage, creates new worker VM clones on `opi5_ssd` by default when that Pimox storage is available, checks that the Pimox bridge already exists, refuses `local` as worker clone storage, and refuses to edit Orange Pi host networking. Keep durable data on the HP Debian laptop's `data-vg` or external backups; treat Orange Pi VM disks as rebuildable worker capacity. `LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to an empty value, so the pipeline owns worker index `1` and VMID `9010` when `LAB_PIMOX_WORKER_COUNT=1`. Set `LAB_PIMOX_SKIP_WORKER_INDEXES=1` if an existing manually created first worker must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID `9010` and VMID `9011`. Pimox workers can join the tailnet during cluster worker bootstrap. Keep the ephemeral or pre-authorized auth key outside Git, either in `~/.config/homelab/tailscale-pimox-worker.authkey` on the Debian host or in `LAB_PIMOX_WORKER_TAILSCALE_AUTH_KEY`. Then run the cluster pipeline with: ```bash LAB_PIMOX_WORKER_TAILSCALE_ENABLED=true ./{{ main_script }} up ``` For existing workers, prefer the Jeannie wrapper because it reads the generated worker keys and exports the correct OpenTofu variables: ```bash ./{{ main_script }} workers tailnet ``` When applying `bootstrap/cluster` directly, use the OpenTofu variable name instead of the Jeannie `LAB_` wrapper: ```bash TF_VAR_worker_tailscale_enabled=true \ tofu -chdir=bootstrap/cluster apply \ -var-file="$PWD/.lab/cluster-workers.auto.tfvars.json" ``` Worker Tailscale defaults to `--accept-routes=false` because these VMs already live on the homelab LAN; accepting an advertised `192.168.100.0/24` route can steal their local LAN route and break SSH or MetalLB traffic. Pod egress SNAT from the Kubernetes pod CIDR to `tailscale0` stays enabled, so pods on Pimox workers can reach tailnet-only services. Set `LAB_PIMOX_WORKER_TAILSCALE_ACCEPT_ROUTES=true` only when the worker must consume tailnet-advertised routes that do not overlap the local LAN. Set `LAB_PIMOX_WORKER_TAILSCALE_POD_EGRESS_SNAT=false` when pod egress through `tailscale0` is not wanted. `./{{ main_script }} doctor-cluster` checks Pimox worker tailnet egress against `${LAB_PIMOX_WORKER_TAILNET_PROBE_HOST:-LAB_DEBIAN_TAILSCALE_IP}` and `${LAB_PIMOX_WORKER_TAILNET_PROBE_PORT:-LAB_GITEA_HTTP_PORT}`. OpenWrt firewall VM automation is available as a standalone command because it attaches to both WAN and LAN bridges. Run `./{{ main_script }} openwrt` after `vmbr1` already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores the VM disk on the selected non-local Pimox storage. It leaves the VM stopped and not enabled for host boot by default. It does not use the Debian Kubernetes golden-node template for OpenWrt. The website and demos images default to `linux/amd64,linux/arm64` because both deployments can schedule on either the Debian host or ARM workers. Override with `WEBSITE_IMAGE_PLATFORMS` or `DEMOS_IMAGE_PLATFORMS` only when intentionally restricting node placement. Build metadata is written under `.lab/` so repeat runs can skip the website or demos image build when the source hash, platform, image reference, and registry manifest still match. The website build is intentionally performed on the Debian homelab path and the Gitea Actions runner, not on a laptop. The current image uses `php:8.5-fpm-alpine`, installs Apache, routes PHP through PHP-FPM, enables OPcache for immutable image deploys, and runs `apps/website/render-static-pages.sh` during the image build. That render step produces static HTML for `/`, `/cv.php`, `/blog.php`, `/demos.php`, and `/homelab-tree.php` in the static languages. Runtime PHP remains for write and translation endpoints such as `save_idea.php`, `visitor_ideas.php`, `translate.php`, and `save_lang.php`. Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the service is already managed manually. The default Gitea target is `jv@192.168.100.73`, install directory `/data/homelab-gitea`, HTTP port `3000`, and SSH port `32222`. Before continuing into Pimox, cluster, apps, or edge changes, `./{{ main_script }} up` runs `./{{ main_script }} preflight`. The preflight verifies Gitea, Debian Docker root, Debian Tailscale IP, RPi Docker root fallback state, RPi Tailscale IP, Pimox worker storage, and OCI edge SSH. Run it directly when changing inventory or host storage/networking: ```bash ./{{ main_script }} preflight ``` If the Debian Docker root check fails after a reinstall, repair it before running the full pipeline: ```bash ./{{ main_script }} fix-debian-docker-root ``` When `./{{ main_script }} up` sees an existing tracked kubeadm control plane but the Kubernetes API is down, it starts the saved runtime before applying cluster changes. Use `./{{ main_script }} rebuild-cluster` only when you intentionally want to destroy and recreate the cluster. ## Validation Useful checks after a rebuild: ```bash export KUBECONFIG=/home/jv/.kube/config kubectl get nodes kubectl top nodes kubectl -n argocd get applications kubectl -n container-registry get pods kubectl -n website-production get pods -o wide kubectl -n demos-static get pods -o wide ssh jv@192.168.100.73 'cd /data/homelab-gitea && sudo docker compose ps' ssh jv@192.168.100.73 'cd /data/homelab-heimdall && sudo docker compose ps' docker info --format '{{.DockerRootDir}}' df -h / /data /data/openebs/local /data/docker ``` The website should be reached through the configured public hostname, not the raw OCI IP address, because the Let's Encrypt certificate is issued for the hostname. Run a read-only health snapshot from the Debian server with: ```bash ./{{ main_script }} help ./{{ main_script }} status ./{{ main_script }} capacity ./{{ main_script }} capacity-advisor ./{{ main_script }} capacity-limits ./{{ main_script }} recover-plan ./{{ main_script }} recover-power --dry-run ./{{ main_script }} impact --since HEAD~1 ./{{ main_script }} review-last-change ./{{ main_script }} scorecard ./{{ main_script }} explain scorecard ./{{ main_script }} gitops-status ./{{ main_script }} cert-check ./{{ main_script }} grafana-dashboards list ./{{ main_script }} release-snapshot ./{{ main_script }} backup-status ./{{ main_script }} synthetic-checks ./{{ main_script }} resource-budget ./{{ main_script }} control-plane status ./{{ main_script }} artifact-cache status ./{{ main_script }} blockchain-devnet status ./{{ main_script }} blockchain-test ./{{ main_script }} blockchain-wallet instructions ./{{ main_script }} golden-ledger check ./{{ main_script }} route-inventory ./{{ main_script }} workers list ./{{ main_script }} change-journal list ./{{ main_script }} map ``` `help` prints the grouped command reference in pipeline order, from inventory and bootstrap through cluster lifecycle, reports, observability, recovery, access, apps, security, AI, and destructive commands. `status` uses the reusable report UI and defaults to showing only failures and warnings so the terminal stays readable. Use `./{{ main_script }} status --all` to show every check, `--details` for row details, `--json` for structured output, or `--verbose` for the older full host/Docker/Kubernetes/Pimox/RPi/ Tailscale/HTTP dump. The cascade includes "what broke" signals such as deployment readiness, pod restarts, node pressure, disk pressure, Traefik 5xx/404 log evidence, and recent Gitea errors. High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node readiness, pod restart storms, unavailable deployments, node/PVC storage pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape coverage. Keep this set small so alerts remain actionable. Application egress is controlled with Kubernetes NetworkPolicies where the lab owns the workload. The website namespace defaults to denied egress and allows only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS for controlled practice targets. `capacity` is the placement report for the hardware you already have. By default it uses the compact report renderer and shows summarized Debian, Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use `./{{ main_script }} capacity --details`, `--json`, `--only problems`, or `--verbose` for the older full capacity dump. `capacity-advisor` interprets capacity, resource-budget, and control-plane placement signals to recommend whether to add Pimox VMs, taint Debian, or fix resource definitions first. `capacity-limits` prints suggested Kubernetes `resources.requests` and `resources.limits` YAML snippets for containers missing them. It uses `infra/resource-recommendations.yml` as a conservative starting profile and does not edit manifests. `recover-plan` prints the disaster recovery order and lightweight prerequisite checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and edge verification. `recover-power` runs the post-outage recovery sequence in dependency order: Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and edge/public checks. Use `recover-power --dry-run` to preview the sequence. `impact` explains what a change may touch before you apply it. Use `./{{ main_script }} impact --since HEAD~1` for the last commit or pass paths like `bootstrap/edge` and `apps/website`. `review-last-change` summarizes the last commit, runs impact analysis for the touched files, and prints validation plus rollback hints. `scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster health, edge health, capacity pressure, security posture, public certificates, inventory, and docs freshness into a compact pass/warn/fail report. `explain` is the read-only follow-up for warnings, failures, and corrective output. It can run safe report commands itself, such as `./{{ main_script }} explain scorecard`, `status`, `capacity`, or `resource-budget`, and it can explain captured output with `./{{ main_script }} explain output < saved-output.txt`. It lists what each finding means, diagnostic commands, whether a Jeannie fix command already exists, and risk/auto-fix safety. Report-oriented commands share the reusable UI module in `scripts/report-ui` and renderer in `scripts/report-render`. Commands emit status TSV rows, and the renderer prints compact grouped summaries by default, with details or JSON available for future TUI/web output. Rows may include separate `fix`, `graph`, and `query` fields so a warning can point to both the next `jeannie` command and the matching Grafana panel or Explore query. `gitops-status` prints a focused Argo CD view: application sync and health, out-of-sync or degraded apps, repository secret presence, recent Argo CD events, and recent repo-server/application-controller errors. `grafana-dashboards` manages repo-provisioned Grafana dashboards under `bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions them through the platform stack; `./{{ main_script }} grafana-dashboards apply` rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional blockchain devnet dashboard uses metrics from `infra/blockchain-devnet/metrics/exporter.py`. `cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory. `release-snapshot` writes a timestamped pre-change report under the homelab state directory with Git state, Kubernetes/Argo CD/Helm state, workload images, public URL status, and pointers to the latest Gitea/OpenTofu backups. `backup-status` reports freshness for Gitea backups, OpenTofu state backups, restore drill reports, and repo-managed Pi-hole restore inputs. The main `status` cascade includes this as a warning signal. `synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API, Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The registry push/pull smoke test is opt-in with `LAB_SYNTHETIC_REGISTRY_PUSH=true`. `resource-budget` checks the report-only policy in `infra/resource-budgets.yml` against Kubernetes pod requests/limits and Debian host disk/memory signals. Use it before adding heavier workloads so the HP laptop remains usable. By default it summarizes missing requests/limits by namespace; use `./{{ main_script }} resource-budget --details` for every pod/container. `control-plane` shows or manages the Debian Kubernetes control-plane scheduling taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint` to keep normal app/data-plane pods off the Debian host while control-plane and DaemonSet workloads continue using their normal tolerations. `artifact-cache` manages an optional Debian-host cache stack in `infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through cache. Use `artifact-cache instructions` for client configuration snippets. `blockchain-devnet` manages an optional local Ethereum Anvil devnet under `infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability learning only; never use real seed phrases or mainnet keys. `blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts with a simple `Counter` contract and ownership tests so there is a known-good baseline before practicing vulnerable contracts. `blockchain-wallet` provides disposable dev-wallet generation, address derivation, and message-signing practice. It is documented in `labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys. `golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the reviewable record of template VMID, storage, OS release, Kubernetes pins, runtime versions, and build metadata for Pimox worker golden images. `route-inventory` reports Kubernetes ingress hosts, paths, backend services, TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and missing monitors are easier to spot. `workers` provides named Kubernetes/Pimox worker lifecycle operations: `list`, `tailnet`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and `rebalance`. Use `recreate-plan ` before replacing one broken Pimox worker so the cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered. `change-journal` lists local journal entries written before risky commands such as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`, `start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch, dirty file count, and command details. `map` prints a dependency map from the canonical inventory, covering public entry, GitOps automation, cluster foundation, DNS, observability, and security learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT. Focused doctor commands run narrower read-only checks. They use the same problems-first UI as `status`; add `--all`, `--details`, `--json`, or `--verbose` when you need more context: ```bash ./{{ main_script }} doctor-edge ./{{ main_script }} doctor-gitea ./{{ main_script }} doctor-rpi ./{{ main_script }} doctor-cluster ``` `doctor-cluster` also checks whether Pimox workers have Tailscale installed, `tailscaled` active, a Tailscale IP, and TCP egress to the configured tailnet probe service. ## RPi Services The Raspberry Pi runs lightweight always-on services outside Kubernetes from `infra/rpi-services`: - Pi-hole on DNS port `53` and web port `8081` - Unbound as Pi-hole's first upstream resolver - Uptime Kuma on port `3001` Deploy or refresh them from the Debian server with: ```bash ./{{ main_script }} rpi-services ``` `./{{ main_script }} up` also runs this step unless `LAB_RPI_SERVICES_DEPLOY=false` is set. Docker data for the Raspberry Pi is expected under `/nvme-storage/docker` when that path is an active writable mount point. If the NVMe is missing or resets, it rewrites Docker's `data-root` to `/var/lib/docker` and restarts Docker so DNS can still come up from the SD/root filesystem. Pi-hole uses Unbound first and public fallback upstreams after that, so DNS can continue resolving if the Unbound container stops. Repo-managed Pi-hole state lives in `infra/rpi-services/adlists.txt`, `infra/rpi-services/local-dns-records.txt`, `infra/rpi-services/cname-records.txt`, and `infra/rpi-services/static-dhcp-hosts.txt`. The bootstrap inserts adlists without deleting manually added Pi-hole lists, then renders the DNS/CNAME/DHCP files into a managed dnsmasq config inside the Pi-hole container. If `PIHOLE_WEBPASSWORD` is not provided, the bootstrap generates one and keeps it in `/opt/homelab-rpi-services/.env`. Uptime Kuma monitor seeds live in `infra/rpi-services/uptime-kuma-monitors.json` and are applied after the first Kuma user exists. `./{{ main_script }} status` and `./{{ main_script }} doctor-rpi` test Pi-hole DNS, direct Unbound DNS, public fallback resolver readiness, Uptime Kuma HTTP, and whether Docker is currently using `/nvme-storage/docker` or has fallen back to `/var/lib/docker`. ## Adding Nodes For Pimox on Orange Pi 5 Plus, `./{{ main_script }} up` can create the Debian 13 arm64 template and worker VM clones automatically when the Orange Pi storage path is healthy. Defaults are intentionally tied to the observed host: Pimox SSH host `192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two 4 GiB worker VMs starting at VMID `9010`, worker clone storage `opi5_ssd`, and no CPU affinity because this Pimox is pinned to Debian Bullseye. The NVMe that was previously attached to the Orange Pi now belongs to the HP Debian laptop as LVM volume group `data-vg`; the current Orange Pi SSD-backed Pimox datastore is `opi5_ssd` and is used for rebuildable VM root disks. Details and override variables are in `bootstrap/provisioning/README.md`. Worker indexes are stable. Index `1` maps to VMID `9010`, node name `pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while allowing higher indexes to be automated. Worker networking defaults to static because the ISP modem DHCP server does not answer Pimox worker VM DHCP requests behind `vmbr0`. Set `LAB_PIMOX_WORKER_NETWORK_MODE=dhcp` only when LAN DHCP is known to answer VM traffic. Static mode uses `LAB_PIMOX_WORKER_STATIC_IPS="192.168.100.66 192.168.100.67 192.168.100.76"`, `LAB_PIMOX_WORKER_GATEWAY=192.168.100.1`, and `LAB_PIMOX_WORKER_DNS_SERVERS="192.168.100.89 1.1.1.1"`. For static workers, Jeannie also adds targeted Pimox-host forwarding exceptions for each worker IP in `DOCKER-USER` when that chain exists, or `FORWARD` otherwise. If `iptables` is missing on the Pimox host, Jeannie installs it before adding those rules. Disable this with `LAB_PIMOX_WORKER_CONFIGURE_FORWARDING=false` only when the Pimox host firewall already permits bridged worker IP traffic. Before adding rules, Jeannie saves `iptables-save` output on the Pimox host under `${LAB_PIMOX_WORKER_IPTABLES_BACKUP_DIR:-~/iptables-backups}`. Worker NIC MAC addresses default to the VMID-derived deterministic format used by the original working Pimox worker path. Set `LAB_PIMOX_WORKER_MAC_MODE=auto` only when you explicitly want Pimox to generate the MAC. Worker creation fails faster than template creation: static guest-network configuration waits up to `LAB_PIMOX_GUEST_AGENT_CONFIG_TIMEOUT_SECONDS=300`, worker IP/SSH readiness waits up to `LAB_PIMOX_WORKER_WAIT_TIMEOUT_SECONDS=600`, and SSH after an IP appears waits up to `LAB_PIMOX_GUEST_SSH_TIMEOUT_SECONDS=600`. Static guest-network validation skips the extra LAN ICMP probe by default because the ISP modem may not answer worker VM pings. Set `LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE=true` only when the gateway/probe host should be required during qemu-guest-agent network setup. After qemu-guest-agent reports an IP, Debian skips the extra ICMP host probe by default and lets SSH be the reachability check. Set `LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE=true` only when you want a faster ICMP gate. `LAB_PIMOX_GUEST_TIMEOUT_SECONDS` remains the broader template-build timeout. Long-running `./{{ main_script }} up` steps keep compact output by default, but the active step line refreshes with elapsed time and the latest log line. Set `JEANNIE_PROGRESS_INTERVAL_SECONDS=5` for faster heartbeats, or `JEANNIE_VERBOSE=true` to stream full command output. Independent host services run through the Go DAG runner during `up`: Gitea deploy/bootstrap, RPi services, Pimox workers, OpenWrt, and OCI edge host checks can progress concurrently. Set `JEANNIE_UP_PARALLELISM=1` to force sequential execution while troubleshooting, or raise it above the default `4` if the lab can absorb more concurrent work. Run a full cluster rebuild from the Debian server with: ```bash ./{{ main_script }} rebuild-cluster ``` That path preserves external Gitea, rebuilds the Pimox template with 2 cores and 4 GiB memory, replaces two Pimox worker VMs with 2 cores and 4 GiB memory, and joins those workers to the Kubernetes cluster. CPU affinity is disabled by default because the Bullseye-pinned Pimox `qm` does not support it. The Raspberry Pi is still included as a Kubernetes worker by default; `nuke` does not clean it unless you explicitly add it to `WORKER_SSH_TARGETS`. To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, PV data, or VM disks, run: ```bash ./{{ main_script }} stop-cluster ``` This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when available, stops any workers listed in `WORKER_SSH_TARGETS`, and gracefully shuts down automated Pimox worker VMs. It intentionally leaves the host `containerd`/Docker stack running so external Docker Compose services such as Gitea stay online. It does not run `kubeadm reset`, delete CNI files, or remove state. Use `LAB_CLUSTER_STOP_CONTAINERD=true` only when you intentionally want to stop the host container runtime too. Use `LAB_CLUSTER_STOP_VM_TIMEOUT_SECONDS` to change the Pimox shutdown timeout, `LAB_CLUSTER_STOP_FORCE=false` to avoid a hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, `LAB_PIMOX_WORKER_BASE_VMID`, and `LAB_PIMOX_SKIP_WORKER_INDEXES` variables to select VM workers. Resume the runtime with: ```bash ./{{ main_script }} start-cluster ``` Kubernetes worker generation defaults to automated Pimox workers only. The Raspberry Pi stays out of the cluster unless `LAB_INCLUDE_RASPBERRY_WORKER=true` is set. Stale manual workers in `.lab/manual-workers.tsv` are ignored unless `LAB_INCLUDE_MANUAL_WORKERS=true` is set. To manage workers manually instead, add entries to `bootstrap/cluster/variables.tf` or a `.tfvars` file: ```hcl worker_nodes = { raspberrypi = { host = "192.168.100.89" user = "jv" node_name = "raspberry" ssh_key_path = "/home/jv/.ssh/id_ed25519" } } ``` Stateful apps currently pin retained local PVs to the `debian` node. Move or duplicate those PV manifests when you want storage on another node. ## Workload Placement `bootstrap/cluster` labels nodes with homelab placement metadata: - `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes` shows `worker` instead of `` in the ROLES column - `homelab.dev/node-role=control-plane`, `homelab.dev/storage=local`, and `homelab.dev/workload-class=control-plane` on the Debian control plane - `homelab.dev/node-role=edge-app`, `homelab.dev/storage=local`, and `homelab.dev/workload-class=edge` on the Raspberry Pi worker - `homelab.dev/node-role=app`, `homelab.dev/storage=ssd`, and `homelab.dev/workload-class=platform` on automated Pimox worker clones when those workers are enabled Override `control_plane_node_labels`, `worker_node_labels`, `LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when the physical layout changes. The current website, demos, and registry manifests are not moved automatically because the public NodePort path and retained OpenEBS hostpath PVs are node-local. Move workloads only after their storage and edge path are ready on the target node. Gitea is outside Kubernetes and is moved by changing the Raspberry Pi Docker install target instead. The stateless platform controllers are pinned to Pimox worker nodes through `homelab.dev/workload-class=platform` and include hostname topology spread plus preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and kube-state-metrics scheduling does not collapse onto the first worker that joins. PVC-backed monitoring StatefulSets are intentionally treated separately because their retained OpenEBS hostpath volumes are node-local. Run `./{{ main_script }} move-prometheus-stack-workers` from the Debian host to label existing worker nodes, destroy only the existing `prometheus-stack` Helm release, delete its retained PVC/PV objects, and recreate the stack on the worker selector when you intentionally accept losing that monitoring data. A planned monitoring data migration should be handled as a separate maintenance task with backup, delete/recreate or storage migration steps, and post-restore checks. The older NodePort path is now reserved for special cases such as the local registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport` support, but app traffic should normally enter through Traefik's MetalLB address instead of per-app NodePorts. Before applying the edge stack, Jeannie configures a Tailscale subnet router so the OCI edge can route to the Traefik LoadBalancer address. The default route advertiser is the Debian control-plane host because it can reach both Tailscale and the MetalLB address; set `LAB_EDGE_TAILSCALE_ROUTER=rpi` to use the Raspberry Pi instead. The cluster stack still supports worker-based subnet-route advertisement when that is useful: ```hcl metallb = { enabled = true repository = "https://metallb.github.io/metallb" version = "0.16.0" namespace = "metallb-system" address_pool = ["192.168.100.240-192.168.100.240"] l2_advertisement_enabled = true pool_name = "homelab-lan" } traefik = { enabled = true repository = "https://helm.traefik.io/traefik" chart_version = "40.2.0" namespace = "traefik" load_balancer_ip = "192.168.100.240" ingress_class = "traefik" } tailscale_subnet_routes = { enabled = true worker_key = "raspberrypi" routes = ["192.168.100.0/24"] } ``` DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for the service hostnames land on the same edge host. For direct LAN testing, point LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at Traefik's address: ```text 192.168.100.240 lab2025.duckdns.org 192.168.100.240 demos.lab2025.duckdns.org 192.168.100.240 n8n.lab2025.duckdns.org 192.168.100.240 prowlarr.lab2025.duckdns.org 192.168.100.240 sonarr.lab2025.duckdns.org 192.168.100.240 radarr.lab2025.duckdns.org 192.168.100.240 qbittorrent.lab2025.duckdns.org 192.168.100.240 kapowarr.lab2025.duckdns.org 192.168.100.240 suwayomi.lab2025.duckdns.org 192.168.100.240 maintainerr.lab2025.duckdns.org 192.168.100.240 grafana.lab2025.duckdns.org 192.168.100.240 argocd.lab2025.duckdns.org ``` Heimdall is external to Kubernetes. For direct LAN testing, use `http://192.168.100.73:8082/`; public access still uses `https://heimdall.lab2025.duckdns.org/` through the OCI edge. The edge stack uses Traefik as its backend by default and validates `http://192.168.100.240:80/` before updating the edge containers. If Tailscale subnet-route approval is not automatic in the tailnet policy, the edge deploy will fail clearly instead of silently keeping a broken public path. For `./{{ main_script }} nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of remote SSH targets when more worker nodes exist. Set it to an empty string for a single-node rebuild. ## Adding Platform Tools Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map. ## Policy Guardrails `bootstrap/platform` installs Kyverno and the upstream baseline Pod Security policies in `Audit` mode. This gives the lab policy reports for unsafe workload settings without blocking existing pods during the first rollout. After reports are clean, individual policies can be promoted to `Enforce` in `bootstrap/platform/main.tf`. `apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for the images built by this repo and pushed to the local registry. The policy targets `192.168.100.73:30500/php-website:*` and `192.168.100.73:30500/demos-static:*`, mutates admitted pods to image digests, and audits whether the image has both: - a valid Cosign signature from the homelab signing key - a signed SPDX SBOM attestation The private Cosign key, generated password, and downloaded Cosign binary live in `${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing identity survives temporary checkout cleanup. Repo-local image state files still live under `.lab/`, which is ignored by Git. `./{{ main_script }} apps` creates or reuses the persistent Cosign key, publishes the public key into the Kyverno namespace as `homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs the pushed images, attaches a signed SPDX SBOM predicate, and verifies both artifacts before refreshing the workload Applications. The policy starts in `Audit` mode so an existing live deployment can recover while the current images are signed; after `./{{ main_script }} apps` verifies both images, change `validationActions` in `apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to `Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to manage the key password externally; otherwise the Debian runner creates `cosign/cosign.password` under the homelab state directory for non-interactive runs. Set `COSIGN_KEY_PREFIX`, `COSIGN_PASSWORD_FILE`, or `COSIGN_BIN` to override those paths. ## DNS Cache `bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with `registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on `169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the rollout compatible with the current kube-proxy iptables path without rewriting kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service CIDR or upstream DNS servers change. ## MetalLB MetalLB is present in `bootstrap/platform` but disabled by default. Enable it only after reserving a LAN IP range outside DHCP and outside any future OpenWrt LAN pool: ```bash export TF_VAR_metallb='{ enabled = true repository = "https://metallb.github.io/metallb" version = "0.16.0" namespace = "metallb-system" address_pool = ["192.168.100.240-192.168.100.250"] l2_advertisement_enabled = true pool_name = "homelab-lan" }' ``` Traefik uses MetalLB for a LAN `LoadBalancer` address. Kubernetes app services such as the website, demos, and n8n should be `ClusterIP` services behind Ingress objects. The local registry remains a `NodePort` because the cluster nodes use it as a pull endpoint. Gitea and Heimdall are not Kubernetes services; they run on the Debian Docker host. ## Secrets Use SOPS with age for secrets that need to live in Git. The Debian host bootstrap installs `age` and `sops`; then run: ```bash ./{{ main_script }} secrets-init ./{{ main_script }} secrets-check ``` `secrets-init` creates `~/.config/sops/age/keys.txt` if needed and renders `.sops.yaml` from `.sops.yaml.example` with the host's public age recipient. Commit `.sops.yaml`, but keep the private age key outside the repo. Operational notes are in `docs/secrets.md`. ## Edge Services The OCI jump box runs the public edge path: ```text nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP ``` Tailnet access policy is tracked in `infra/tailscale/tailnet-policy.hujson`. Validate it before applying with: ```bash ./{{ main_script }} tailnet-policy-check ``` The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in `bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars` file when the public host, SSH key, server name, backend Tailscale IP, or Traefik backend address changes. The `/git/` route is intentionally different from the Kubernetes app routes: it proxies to Gitea on the Debian host through Tailscale instead of Traefik. This keeps public read-only source browsing available even when the cluster has been destroyed. For the main website, nginx terminates TLS, serves cached HTML and static assets, compresses text responses with gzip, advertises HTTP/2, reuses TLS sessions, and keeps upstream connections open to the Traefik backend. The application side marks pre-rendered HTML as `Cache-Control: public, max-age=300, s-maxage=3600`; the edge keeps CSS and JS under the stronger immutable asset policy. In local measurements, cached single-resource requests are dominated by the remote TCP/TLS path, while a reused HTTP/2 asset batch completes in roughly 60 ms after the connection is established. Use the configured `server_name` in the browser, for example `https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser certificate warning because the trusted certificate is issued for the hostname. The edge stack uses HTTP-01 validation and requests one certificate covering `server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to receive expiry notices, or leave it empty to register without an email. Set `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. Operational runbooks: - [Edge failures](docs/runbooks/edge-failures.md) - [Gitea failures](docs/runbooks/gitea-failures.md) - [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) ## Adding Apps Add Kubernetes manifests under `apps/` and register them in `bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and self-healing for the app. The `heimdall` Argo CD app name is retained as a small route bundle for Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard itself. The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is exposed at `https://n8n.lab2025.duckdns.org`. It receives `OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing Debian-host Ollama API for translation prewarming and batch jobs. The ARR/media stack is intentionally not deployed by the Kubernetes app pipeline. It is better suited to a Docker Compose stack on the Debian host with persistent volumes under `/data`, where large media paths, qBittorrent writes, and service config directories can survive cluster rebuilds without OpenEBS node-affinity or PVC migration concerns. The active host-level Compose definition lives under `infra/arr-stack` and stores persistent data under `/data/arr`. Historical Kubernetes manifests are archived under `archive/apps/arr-stack` for migration reference only. ## Storage OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as `/data/openebs/local/registry`; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories. For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath data still lives under `/data/openebs/local`, and larger service data such as Gitea, Ollama models, and app volumes should stay under `/data`. ## Gitea Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt. The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. Public source browsing stays available through `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account. ## Heimdall Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy. Deploy or refresh it from the Debian server with: ```bash ./{{ main_script }} deploy-heimdall ``` `./{{ main_script }} up` also runs this step during the independent host services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar container that upserts repo-managed tiles without deleting manually added ones. The Debian bare repo remains the GitOps mirror: ```text /home/jv/git-server/my-homelab-configs.git ``` Argo CD consumes that Debian mirror through the default `gitops_repo_url`. Gitea Actions pushes the `main` commit into the mirror before running the selected deploy command. The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports `knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash ./{{ main_script }} deploy-gitea ``` ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.