Homelab Kubernetes Pipeline

This repo bootstraps a hybrid kubeadm cluster and then hands app delivery to Argo CD.

Architecture

The lab is intentionally small but production-shaped:

• a Debian amd64 host runs the kubeadm control plane and local deployment tools
• a Raspberry Pi arm64 node runs selected workloads
• the Raspberry Pi also runs the always-on Gitea Docker service outside Kubernetes
• the Debian host keeps a bare GitOps mirror under /home/jv/git-server/my-homelab-configs.git
• a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker templates
• OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge
• Argo CD continuously reconciles Kubernetes manifests from this repo
• a local registry stores the website and demos images built for the worker architecture
• SOPS with age is the committed secret-management path for future encrypted Kubernetes secrets
• an OCI jump box provides the public edge path back into the homelab over Tailscale

Run ./lab.sh up and ./lab.sh nuke only from the Debian homelab server. The script intentionally refuses to run from non-Debian machines so a laptop cannot accidentally modify the cluster.

Flow

1. bootstrap/provisioning

   • prepares a Debian server as a PXE and preseed service for arm64 VMs
   • serves Debian 13 arm64 netboot assets through TFTP and HTTP
   • creates a golden image install path with Kubernetes, containerd, qemu-guest-agent, cloud-init, and storage client packages ready
   • is driven by ./lab.sh up when Pimox is reachable, without changing Orange Pi host networking

2. bootstrap/cluster

   • creates the kubeadm control plane on the Debian amd64 node
   • joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes
   • configures Calico-compatible pod CIDR
   • configures containerd to pull from the in-cluster NodePort registry
   • creates retained host directories under /var/openebs/local

3. bootstrap/platform

   • installs a minimal Calico deployment through the Tigera operator
   • installs NodeLocal DNSCache for node-local DNS query caching
   • installs MetalLB for LAN LoadBalancer services
   • installs Traefik as the single Kubernetes ingress entry point
   • installs OpenEBS
   • creates openebs-hostpath-retain
   • installs Argo CD
   • installs Kyverno with audit-first baseline Pod Security policies
   • registers the private GitOps repo without storing the SSH private key in Terraform state

4. bootstrap/apps

   • registers Argo CD Applications from the applications map
   • passes the website image produced by the build step into Argo CD as a Kustomize image override
   • default apps are container-registry, website-production, demos-static, and heimdall

5. bootstrap/edge

   • connects to the OCI jump box
   • uploads nginx, HAProxy, Varnish, and Squid configs
   • obtains and renews Let's Encrypt certificates for the configured hostname
   • runs the edge cache/proxy chain with Docker Compose

Prerequisites

On the Debian host:

• OpenTofu
• Docker with Buildx
• kubeadm, kubelet, kubectl, and containerd
• SSH access to worker nodes
• SSH access to the OCI edge host
• enough persistent storage for /var/openebs/local and /var/lib/docker

The default kubeconfig path is /home/jv/.kube/config. Override it with KUBECONFIG_PATH or TF_VAR_kubeconfig_path when needed.

Deploying

From the Debian server:

cd ~/my-homelab-configs
./lab.sh up

The script first deploys external Gitea to the Raspberry Pi with Docker Compose so Git stays outside the Kubernetes rebuild blast radius. It then detects the Pimox host at 192.168.100.80 in auto mode. When SSH, qm, and vmbr0 are available, it applies bootstrap/provisioning, creates or reuses the Debian 13 arm64 template, creates or reuses one worker VM clone, discovers the guest IP through qemu-guest-agent, and passes that worker into the cluster layer. It then applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the local registry, builds the website and demos images when their source changed, pushes them to the registry, recreates pods only after a new image is built, and applies the edge stack.

Set LAB_PIMOX_PIPELINE=false to skip Pimox automation. Set LAB_PIMOX_WORKER_COUNT=0 to create or refresh only the template. The pipeline keeps the template on its configured local storage, creates new worker VM clones on nvme_thin_pool by default, checks that the Pimox bridge already exists, refuses local as worker clone storage, and refuses to edit Orange Pi host networking.

LAB_PIMOX_SKIP_WORKER_INDEXES defaults to an empty value, so the pipeline owns worker index 1 and VMID 9010 when LAB_PIMOX_WORKER_COUNT=1. Set LAB_PIMOX_SKIP_WORKER_INDEXES=1 if an existing manually created first worker must be left untouched, or set LAB_PIMOX_WORKER_COUNT=2 to manage both VMID 9010 and VMID 9011.

OpenWrt firewall VM automation is available as a standalone command because it attaches to both WAN and LAN bridges. Run ./lab.sh openwrt after vmbr1 already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, imports it as VM 9100, attaches vmbr0 as WAN and vmbr1 as LAN, and stores the VM disk on nvme_thin_pool. It leaves the VM stopped and not enabled for host boot by default. It does not use the Debian Kubernetes golden-node template for OpenWrt.

The website and demos images default to linux/arm64 because both deployments are pinned to the Raspberry Pi worker. Override with WEBSITE_IMAGE_PLATFORMS or DEMOS_IMAGE_PLATFORMS only if node placement changes.

Build metadata is written under .lab/ so repeat runs can skip the website or demos image build when the source hash, platform, image reference, and registry manifest still match.

Set LAB_GITEA_DEPLOY=false to skip the external Gitea deployment step when the Raspberry Pi service is already managed manually. The default Gitea target is jv@192.168.100.89, install directory /opt/homelab-gitea, HTTP port 3000, and SSH port 32222.

Validation

Useful checks after a rebuild:

export KUBECONFIG=/home/jv/.kube/config

kubectl get nodes
kubectl -n argocd get applications
kubectl -n container-registry get pods
kubectl -n website-production get pods -o wide
kubectl -n demos-static get pods -o wide
kubectl -n heimdall get pods -o wide

ssh jv@192.168.100.89 'cd /opt/homelab-gitea && sudo docker compose ps'

docker info --format '{{.DockerRootDir}}'
df -h / /var/openebs/local /var/lib/docker

The website should be reached through the configured public hostname, not the raw OCI IP address, because the Let's Encrypt certificate is issued for the hostname.

Adding Nodes

For Pimox on Orange Pi 5 Plus, ./lab.sh up can create the Debian 13 arm64 template and worker VM clones automatically. Defaults are intentionally tied to the observed host: Pimox SSH host 192.168.100.80, bridge vmbr0, template VMID 9000 on local storage, two 4 GiB worker VMs starting at VMID 9010, worker clone storage nvme_thin_pool, and no CPU affinity because this Pimox is pinned to Debian Bullseye. Details and override variables are in bootstrap/provisioning/README.md.

Worker indexes are stable. Index 1 maps to VMID 9010, node name pimox-worker-01, and worker key pimox01; index 2 maps to VMID 9011, and so on. LAB_PIMOX_SKIP_WORKER_INDEXES=1 leaves the first slot unmanaged while allowing higher indexes to be automated.

Run a full cluster rebuild from the Debian server with:

./lab.sh rebuild-cluster

That path preserves external Raspberry Pi Gitea, rebuilds the Pimox template with 2 cores and 4 GiB memory, replaces two Pimox worker VMs with 2 cores and 4 GiB memory, and joins those workers to the Kubernetes cluster. CPU affinity is disabled by default because the Bullseye-pinned Pimox qm does not support it. The Raspberry Pi is still included as a Kubernetes worker by default; nuke does not clean it unless you explicitly add it to WORKER_SSH_TARGETS, so the external Gitea Docker service survives cluster rebuilds.

To exclude the Raspberry Pi from the Kubernetes cluster, set LAB_INCLUDE_RASPBERRY_WORKER=false. To manage workers manually instead, add entries to bootstrap/cluster/variables.tf or a .tfvars file:

worker_nodes = {
  raspberrypi = {
    host         = "192.168.100.89"
    user         = "jv"
    node_name    = "raspberry"
    ssh_key_path = "/home/jv/.ssh/id_ed25519"
  }
}

Stateful apps currently pin retained local PVs to the debian node. Move or duplicate those PV manifests when you want storage on another node.

Workload Placement

bootstrap/cluster labels nodes with homelab placement metadata:

• node-role.kubernetes.io/worker=worker on every worker so kubectl get nodes shows worker instead of <none> in the ROLES column
• homelab.dev/node-role=control-plane, homelab.dev/storage=local, and homelab.dev/workload-class=control-plane on the Debian control plane
• homelab.dev/node-role=edge-app, homelab.dev/storage=local, and homelab.dev/workload-class=edge on the Raspberry Pi worker
• homelab.dev/node-role=app, homelab.dev/storage=nvme, and homelab.dev/workload-class=platform on automated Pimox worker clones

Override control_plane_node_labels, worker_node_labels, LAB_RASPBERRY_NODE_LABELS_JSON, or LAB_PIMOX_WORKER_NODE_LABELS_JSON when the physical layout changes. The current website, demos, and registry manifests are not moved automatically because the public NodePort path and retained OpenEBS hostpath PVs are node-local. Move workloads only after their storage and edge path are ready on the target node. Gitea is outside Kubernetes and is moved by changing the Raspberry Pi Docker install target instead.

The stateless platform controllers are pinned to Pimox worker nodes through homelab.dev/workload-class=platform and include hostname topology spread plus preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and kube-state-metrics scheduling does not collapse onto the first worker that joins. PVC-backed monitoring StatefulSets are intentionally treated separately because their retained OpenEBS hostpath volumes are node-local. Run ./lab.sh move-prometheus-stack-workers from the Debian host to label existing worker nodes, destroy only the existing prometheus-stack Helm release, delete its retained PVC/PV objects, and recreate the stack on the worker selector when you intentionally accept losing that monitoring data. A planned monitoring data migration should be handled as a separate maintenance task with backup, delete/recreate or storage migration steps, and post-restore checks.

The older NodePort path is now reserved for special cases such as the local registry. bootstrap/cluster still contains homelab-tailscale-nodeport support, but app traffic should normally enter through Traefik's MetalLB address instead of per-app NodePorts. The cluster stack advertises the LAN subnet from the configured Tailscale worker so the OCI edge can route to the Traefik LoadBalancer address:

metallb = {
  enabled                  = true
  repository               = "https://metallb.github.io/metallb"
  version                  = "0.16.0"
  namespace                = "metallb-system"
  address_pool             = ["192.168.100.240-192.168.100.240"]
  l2_advertisement_enabled = true
  pool_name                = "homelab-lan"
}

traefik = {
  enabled          = true
  repository       = "https://helm.traefik.io/traefik"
  chart_version    = "40.2.0"
  namespace        = "traefik"
  load_balancer_ip = "192.168.100.240"
  ingress_class    = "traefik"
}

tailscale_subnet_routes = {
  enabled    = true
  worker_key = "raspberrypi"
  routes     = ["192.168.100.0/24"]
}

DuckDNS resolves *.lab2025.duckdns.org to the OCI edge, so public requests for the service hostnames land on the same edge host. For direct LAN testing, point LAN DNS, /etc/hosts, or a Tailscale DNS override for app hostnames at Traefik's address:

192.168.100.240 lab2025.duckdns.org
192.168.100.240 demos.lab2025.duckdns.org
192.168.100.240 heimdall.lab2025.duckdns.org
192.168.100.240 grafana.lab2025.duckdns.org
192.168.100.240 argocd.lab2025.duckdns.org

The edge stack uses Traefik as its backend by default and validates http://192.168.100.240:80/ before updating the edge containers. If Tailscale subnet-route approval is not automatic in the tailnet policy, the edge deploy will fail clearly instead of silently keeping a broken public path.

For ./lab.sh nuke, set WORKER_SSH_TARGETS to a space-separated list of remote SSH targets when more worker nodes exist. Set it to an empty string for a single-node rebuild.

Adding Platform Tools

Add Helm releases through bootstrap/platform's extra_helm_releases map.

Policy Guardrails

bootstrap/platform installs Kyverno and the upstream baseline Pod Security policies in Audit mode. This gives the lab policy reports for unsafe workload settings without blocking existing pods during the first rollout. After reports are clean, individual policies can be promoted to Enforce in bootstrap/platform/main.tf.

DNS Cache

bootstrap/platform installs NodeLocal DNSCache in kube-system with registry.k8s.io/dns/k8s-dns-node-cache. The default listens on 169.254.20.10 and the kube-dns service IP 10.96.0.10, which keeps the rollout compatible with the current kube-proxy iptables path without rewriting kubelet DNS settings across the nodes. Override nodelocal_dns if the service CIDR or upstream DNS servers change.

MetalLB

MetalLB is present in bootstrap/platform but disabled by default. Enable it only after reserving a LAN IP range outside DHCP and outside any future OpenWrt LAN pool:

export TF_VAR_metallb='{
  enabled = true
  repository = "https://metallb.github.io/metallb"
  version = "0.16.0"
  namespace = "metallb-system"
  address_pool = ["192.168.100.240-192.168.100.250"]
  l2_advertisement_enabled = true
  pool_name = "homelab-lan"
}'

Traefik uses MetalLB for a LAN LoadBalancer address. App services such as the website, demos, and Heimdall should be ClusterIP services behind Kubernetes Ingress objects. The local registry remains a NodePort because the cluster nodes use it as a pull endpoint. Gitea is not a Kubernetes service; it runs on the Raspberry Pi Docker host.

Secrets

Use SOPS with age for secrets that need to live in Git. Start from .sops.yaml.example, replace the age recipient with the public key generated on the Debian host, and commit the resulting .sops.yaml. Keep the private age key outside the repo. Operational notes are in docs/secrets.md.

Edge Services

The OCI jump box runs the public edge path:

nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP

The bootstrap/edge stack renders configs from bootstrap/edge/templates and deploys them to /opt/homelab-edge on the OCI host. Defaults are in bootstrap/edge/variables.tf; override them through TF_VAR_* or a .tfvars file when the public host, SSH key, server name, backend Tailscale IP, or Traefik backend address changes.

The /git/ route is intentionally different from the Kubernetes app routes: it proxies to Gitea on the Raspberry Pi at the configured gitea_backend_port instead of Traefik. This keeps public read-only source browsing available even when the cluster has been destroyed.

Use the configured server_name in the browser, for example https://lab2025.duckdns.org. A raw OCI IP address will still show a browser certificate warning because the trusted certificate is issued for the hostname.

The edge stack uses HTTP-01 validation and requests one certificate covering server_name plus additional_server_names. DuckDNS resolves sub-subdomains under lab2025.duckdns.org to the same edge IP, so inbound TCP 80 and 443 must be open before ./lab.sh up runs. Set TF_VAR_letsencrypt_email to receive expiry notices, or leave it empty to register without an email. Set TF_VAR_enable_letsencrypt=false to keep using the temporary local certificate.

Adding Apps

Add Kubernetes manifests under apps/<name> and register them in bootstrap/apps's applications map. Argo CD will own sync, pruning, and self-healing for the app.

The heimdall app is intentionally waited on at the end of ./lab.sh apps. It runs the LinuxServer.io Heimdall dashboard, persists /config on OpenEBS, and seeds tiles for the website, demo apps, Gitea, Grafana, Prometheus, Alertmanager, Argo CD, the local registry, and Heimdall itself. Because Heimdall does not support reverse-proxy subfolder hosting cleanly, it is exposed through the dedicated hostname heimdall.lab2025.duckdns.org rather than a /heimdall/ path.

Storage

OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as /var/openebs/local/registry; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories.

For the current lab, /var/openebs/local and /var/lib/docker are expected to live on larger storage than the root filesystem. This keeps retained PVs, container layers, Buildx state, and image caches from filling /.

Gitea

Gitea is external bootstrap infrastructure. It runs on the Raspberry Pi as an always-on Docker Compose service from infra/gitea/docker-compose.yml, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt.

The default data path is /opt/homelab-gitea/data on the Raspberry Pi SD card. That is acceptable for the current temporary setup; move LAB_GITEA_INSTALL_DIR to an SSD mount when the SSD is added.

Public source browsing stays available through https://lab2025.duckdns.org/git/. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account.

The Debian bare repo remains the GitOps mirror:

/home/jv/git-server/my-homelab-configs.git

Argo CD consumes that Debian mirror through the default gitops_repo_url. Gitea Actions pushes the main commit into the mirror before running the selected deploy command.

The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports knownhosts: key is unknown after the Debian host was rebuilt or its SSH host key changed, refresh argocd-ssh-known-hosts-cm in the argocd namespace, restart argocd-repo-server, and hard-refresh the affected Application.

Deploy or refresh the external Gitea container from the Debian host with:

./lab.sh deploy-gitea

Gitea Backups

./lab.sh up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the Raspberry Pi, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Raspberry Pi Gitea data directory.

Run a manual backup from the Debian server with:

./lab.sh backup-gitea

Run the restore drill manually with:

./lab.sh drill-gitea-restore

Useful checks:

systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs only on pushes to main and targets a repository-scoped Debian host runner with the label homelab-debian.

The workflow only blocks automatic deploy for Raspberry Pi Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in lab.sh. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./lab.sh apps when the Debian runner already has a cluster kubeconfig, or ./lab.sh rebuild-cluster when it does not.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./lab.sh bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.89:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./lab.sh install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./lab.sh up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./lab.sh nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /var/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so the Raspberry Pi Gitea host is not cleaned unless you explicitly include it.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Generated runtime language JSON is saved through save_lang.php on the website PVC.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./lab.sh up builds and pushes two independent images:

• a content-hash php-website tag generated by lab.sh and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, lab.sh hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable.

After ./lab.sh apps, the live deployment image should be a content-hash tag, for example 192.168.100.68:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from the Raspberry Pi worker without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. With the current deployments pinned to the Raspberry Pi, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.