# Application Operations ## Adding Apps Add Kubernetes manifests under `apps/` and register them in `bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and self-healing for the app. The `heimdall` Argo CD app name is retained as a small route bundle for Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes workloads there. Use `./{{ main_script }} deploy-heimdall` for the dashboard itself. The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is exposed at `https://n8n.lab2025.duckdns.org`. It receives `OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing Debian-host Ollama API for translation prewarming and batch jobs. The ARR/media stack is intentionally not deployed by the Kubernetes app pipeline. It is better suited to a Docker Compose stack on the Debian host with persistent volumes under `/data`, where large media paths, qBittorrent writes, and service config directories can survive cluster rebuilds without OpenEBS node-affinity or PVC migration concerns. The active host-level Compose definition lives under `infra/arr-stack` and stores persistent data under `/data/arr`. Historical Kubernetes manifests are archived under `archive/apps/arr-stack` for migration reference only. ## Storage OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as `/data/openebs/local/registry`; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories. For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath data still lives under `/data/openebs/local`, and larger service data such as Gitea, Ollama models, and app volumes should stay under `/data`. ## Gitea Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt. The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. Public source browsing stays available through `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account. ## Heimdall Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy. Deploy or refresh it from the Debian server with: ```bash ./{{ main_script }} deploy-heimdall ``` `./{{ main_script }} up` also runs this step during the independent host services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar container that upserts repo-managed tiles without deleting manually added ones. The Debian bare repo remains the GitOps mirror: ```text /home/jv/git-server/my-homelab-configs.git ``` Argo CD consumes that Debian mirror through the default `gitops_repo_url`. Gitea Actions pushes the `main` commit into the mirror before running the selected deploy command. The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports `knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash ./{{ main_script }} deploy-gitea ``` ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Storage OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as `/data/openebs/local/registry`; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories. For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath data still lives under `/data/openebs/local`, and larger service data such as Gitea, Ollama models, and app volumes should stay under `/data`. ## Gitea Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt. The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. Public source browsing stays available through `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account. ## Heimdall Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy. Deploy or refresh it from the Debian server with: ```bash ./{{ main_script }} deploy-heimdall ``` `./{{ main_script }} up` also runs this step during the independent host services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar container that upserts repo-managed tiles without deleting manually added ones. The Debian bare repo remains the GitOps mirror: ```text /home/jv/git-server/my-homelab-configs.git ``` Argo CD consumes that Debian mirror through the default `gitops_repo_url`. Gitea Actions pushes the `main` commit into the mirror before running the selected deploy command. The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports `knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash ./{{ main_script }} deploy-gitea ``` ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Gitea Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt. The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe. Public source browsing stays available through `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account. ## Heimdall Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy. Deploy or refresh it from the Debian server with: ```bash ./{{ main_script }} deploy-heimdall ``` `./{{ main_script }} up` also runs this step during the independent host services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar container that upserts repo-managed tiles without deleting manually added ones. The Debian bare repo remains the GitOps mirror: ```text /home/jv/git-server/my-homelab-configs.git ``` Argo CD consumes that Debian mirror through the default `gitops_repo_url`. Gitea Actions pushes the `main` commit into the mirror before running the selected deploy command. The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports `knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash ./{{ main_script }} deploy-gitea ``` ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Heimdall Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy. Deploy or refresh it from the Debian server with: ```bash ./{{ main_script }} deploy-heimdall ``` `./{{ main_script }} up` also runs this step during the independent host services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`. Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar container that upserts repo-managed tiles without deleting manually added ones. The Debian bare repo remains the GitOps mirror: ```text /home/jv/git-server/my-homelab-configs.git ``` Argo CD consumes that Debian mirror through the default `gitops_repo_url`. Gitea Actions pushes the `main` commit into the mirror before running the selected deploy command. The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports `knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace, restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash ./{{ main_script }} deploy-gitea ``` ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Gitea Backups `./{{ main_script }} up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default retention is 30 days. The same install step also creates `homelab-gitea-restore-drill.timer`. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under `/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash ./{{ main_script }} backup-gitea ``` Run the restore drill manually with: ```bash ./{{ main_script }} drill-gitea-restore ./{{ main_script }} drill-restore ./{{ main_script }} drill-pihole-restore ``` `drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes. Useful checks: ```bash ./{{ main_script }} backup-status systemctl list-timers homelab-gitea-backup.timer systemctl list-timers homelab-gitea-restore-drill.timer sudo systemctl start homelab-gitea-backup.service ls -lh /home/jv/backups/gitea ls -lh /home/jv/backups/gitea-restore-drills ``` ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Gitea Actions This repo includes a Gitea Actions workflow at `.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and `main`, and deploys only from `main`. That keeps the promotion path simple: ```text dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync ``` The workflow targets a repository-scoped Debian host runner with the label `homelab-debian`. Run the same validation locally before promoting: ```bash ./{{ main_script }} validate ``` ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Identity and access audit Run a read-only access inventory from the Debian server with: ```bash ./{{ main_script }} access-audit ``` The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities: - admin kubeconfig only on the Debian control host - read-only kubeconfig for dashboards and audits - separate service accounts for automation - explicit SSH inventory for Debian, RPi, Pimox, and OCI edge - repo-managed Tailscale ACLs The read-only Kubernetes identity is managed in `apps/access-control`. After that app syncs, generate a separate read-only kubeconfig on the Debian host: ```bash ./{{ main_script }} kubeconfig-readonly ``` The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` functions in `{{ main_script }}`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by default: Actions runs `./{{ main_script }} doctor-versions` when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs `./{{ main_script }} apps`; if kubelet minor drift is detected it runs `./{{ main_script }} rebuild-cluster`; if no kubeconfig exists it also runs `./{{ main_script }} rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. `./{{ main_script }} bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from `HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or `/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea. Enable Actions for the repository in Gitea, then create a repository-level runner token from: ```text https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners ``` Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs GITEA_RUNNER_REGISTRATION_TOKEN='' ./{{ main_script }} install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state. The deployment job is non-interactive. User `jv` must be able to run `sudo -n true` on the Debian host for deployment commands that require sudo. Useful checks: ```bash systemctl status homelab-gitea-runner.service journalctl -u homelab-gitea-runner.service -n 100 --no-pager ``` ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Renovate `renovate.json` defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run `./{{ main_script }} up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds `./{{ main_script }} nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. ## Website App The website is a PHP app under `apps/website`. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in `apps/website/lang`; `en.php` and `nah.php` are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin `/translate.php` endpoint, which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the `website-production` namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom `website-translator` Ollama model defined in `apps/website/ollama/Modelfile`. Generated runtime language JSON is saved through `save_lang.php` on the website PVC, and `translate.php` emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts. Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash ./{{ main_script }} website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash ./{{ main_script }} website-ollama-listen ``` The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama` is true: it installs the service, stores models under `/data/ollama/models`, binds the API to the configured LAN listener, and pulls the lightweight `ai_gateway.model`. On an existing host, apply just this setup with: ```bash ./{{ main_script }} ollama-setup ``` `{{ main_script }}` also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when `ai_gateway.enabled` is true in `homelab.yml`, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is: ```bash ollama pull qwen2.5:0.5b ``` Disable it for low-CPU sessions with: ```bash LAB_BACKSTAGE_BRAIN_ENABLED=false ./{{ main_script }} doctor-edge ``` Build the local homelab knowledge index separately from the main deployment: ```bash ./{{ main_script }} ai-index ./{{ main_script }} ai-check ./{{ main_script }} ask "how do I check edge to Gitea?" ``` The index is built from the non-secret source list in `infra/ai/knowledge-sources.txt`, skips paths in `infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use `ask` as the runbook and command finder. It returns the closest indexed docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval. The CV page has two client-side presentation modes: - `Elegant`: dark, minimal, terminal-inspired styling with a square profile image and light green console text. - `Fancy`: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect. The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. `./{{ main_script }} up` builds and pushes two independent images: - a content-hash `php-website` tag generated by `{{ main_script }}` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `{{ main_script }}` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. After `./{{ main_script }} apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository access before restarting pods. The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads. The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the `demos-static` pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus. Current demo inventory: - Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video. - Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version. - Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering. - Architecture simulator: click-driven load, crash, and auto-scale simulation. - Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions. - Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path. - Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path. - Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by the app image build and with the image globs in `apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.