my-homelab-configs/docs/app-ops.md

Application Operations

Adding Apps

Add Kubernetes manifests under apps/<name> and register them in bootstrap/apps's applications map. Argo CD will own sync, pruning, and self-healing for the app.

The heimdall Argo CD app name is retained as a small route bundle for Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes workloads there. Use ./jeannie deploy-heimdall for the dashboard itself.

The n8n app runs in the n8n namespace with retained OpenEBS storage and is exposed at https://n8n.lab2025.duckdns.org. It receives OLLAMA_BASE_URL=http://192.168.100.73:11434 so workflows can call the existing Debian-host Ollama API for translation prewarming and batch jobs.

The ARR/media stack is intentionally not deployed by the Kubernetes app pipeline. It is better suited to a Docker Compose stack on the Debian host with persistent volumes under /data, where large media paths, qBittorrent writes, and service config directories can survive cluster rebuilds without OpenEBS node-affinity or PVC migration concerns. The active host-level Compose definition lives under infra/arr-stack and stores persistent data under /data/arr. Historical Kubernetes manifests are archived under archive/apps/arr-stack for migration reference only.

Storage

OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as /data/openebs/local/registry; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories.

For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the standard Docker root /var/lib/docker is acceptable. OpenEBS retained hostpath data still lives under /data/openebs/local, and larger service data such as Gitea, Ollama models, and app volumes should stay under /data.

Gitea

Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/gitea/docker-compose.yml, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt.

The default data path is /data/homelab-gitea/data on the HP laptop NVMe. Docker may use the standard /var/lib/docker root because / is also on NVMe.

Public source browsing stays available through https://lab2025.duckdns.org/git/. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account.

Heimdall

Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/heimdall/docker-compose.yml, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy.

Deploy or refresh it from the Debian server with:

./jeannie deploy-heimdall

./jeannie up also runs this step during the independent host services stage unless LAB_HEIMDALL_DEPLOY=false is set. The default data path is /data/homelab-heimdall/data/config, and the local HTTP port is 8082. Dashboard links are seeded from infra/heimdall/links.json by a sidecar container that upserts repo-managed tiles without deleting manually added ones.

The Debian bare repo remains the GitOps mirror:

/home/jv/git-server/my-homelab-configs.git

Argo CD consumes that Debian mirror through the default gitops_repo_url. Gitea Actions pushes the main commit into the mirror before running the selected deploy command.

The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports knownhosts: key is unknown after the Debian host was rebuilt or its SSH host key changed, refresh argocd-ssh-known-hosts-cm in the argocd namespace, restart argocd-repo-server, and hard-refresh the affected Application.

Deploy or refresh the external Gitea container from the Debian host with:

./jeannie deploy-gitea

Gitea Backups

./jeannie up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the configured Gitea host, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Gitea data directory.

Run a manual backup from the Debian server with:

./jeannie backup-gitea

Run the restore drill manually with:

./jeannie drill-gitea-restore
./jeannie drill-restore
./jeannie drill-pihole-restore

drill-restore validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes.

Useful checks:

./jeannie backup-status
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Storage

OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use retained local PV paths such as /data/openebs/local/registry; these paths are intentionally outside kubeadm reset paths so data can survive cluster destroy/create cycles. Those critical volumes are declared explicitly as retained local PVs so a rebuilt cluster binds back to the same host paths instead of creating fresh directories.

For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the standard Docker root /var/lib/docker is acceptable. OpenEBS retained hostpath data still lives under /data/openebs/local, and larger service data such as Gitea, Ollama models, and app volumes should stay under /data.

Gitea

Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/gitea/docker-compose.yml, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt.

The default data path is /data/homelab-gitea/data on the HP laptop NVMe. Docker may use the standard /var/lib/docker root because / is also on NVMe.

Public source browsing stays available through https://lab2025.duckdns.org/git/. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account.

Heimdall

Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/heimdall/docker-compose.yml, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy.

Deploy or refresh it from the Debian server with:

./jeannie deploy-heimdall

./jeannie up also runs this step during the independent host services stage unless LAB_HEIMDALL_DEPLOY=false is set. The default data path is /data/homelab-heimdall/data/config, and the local HTTP port is 8082. Dashboard links are seeded from infra/heimdall/links.json by a sidecar container that upserts repo-managed tiles without deleting manually added ones.

The Debian bare repo remains the GitOps mirror:

/home/jv/git-server/my-homelab-configs.git

Argo CD consumes that Debian mirror through the default gitops_repo_url. Gitea Actions pushes the main commit into the mirror before running the selected deploy command.

The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports knownhosts: key is unknown after the Debian host was rebuilt or its SSH host key changed, refresh argocd-ssh-known-hosts-cm in the argocd namespace, restart argocd-repo-server, and hard-refresh the affected Application.

Deploy or refresh the external Gitea container from the Debian host with:

./jeannie deploy-gitea

Gitea Backups

./jeannie up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the configured Gitea host, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Gitea data directory.

Run a manual backup from the Debian server with:

./jeannie backup-gitea

Run the restore drill manually with:

./jeannie drill-gitea-restore
./jeannie drill-restore
./jeannie drill-pihole-restore

drill-restore validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes.

Useful checks:

./jeannie backup-status
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Gitea

Gitea is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/gitea/docker-compose.yml, not as a Kubernetes workload. This keeps Git available when the Kubernetes cluster is destroyed and rebuilt.

The default data path is /data/homelab-gitea/data on the HP laptop NVMe. Docker may use the standard /var/lib/docker root because / is also on NVMe.

Public source browsing stays available through https://lab2025.duckdns.org/git/. Registration is disabled and anonymous users can view public repositories, so the blog can link to code read-only while writes still require an authenticated Gitea account.

Heimdall

Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/heimdall/docker-compose.yml, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy.

Deploy or refresh it from the Debian server with:

./jeannie deploy-heimdall

./jeannie up also runs this step during the independent host services stage unless LAB_HEIMDALL_DEPLOY=false is set. The default data path is /data/homelab-heimdall/data/config, and the local HTTP port is 8082. Dashboard links are seeded from infra/heimdall/links.json by a sidecar container that upserts repo-managed tiles without deleting manually added ones.

The Debian bare repo remains the GitOps mirror:

/home/jv/git-server/my-homelab-configs.git

Argo CD consumes that Debian mirror through the default gitops_repo_url. Gitea Actions pushes the main commit into the mirror before running the selected deploy command.

The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports knownhosts: key is unknown after the Debian host was rebuilt or its SSH host key changed, refresh argocd-ssh-known-hosts-cm in the argocd namespace, restart argocd-repo-server, and hard-refresh the affected Application.

Deploy or refresh the external Gitea container from the Debian host with:

./jeannie deploy-gitea

Gitea Backups

./jeannie up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the configured Gitea host, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Gitea data directory.

Run a manual backup from the Debian server with:

./jeannie backup-gitea

Run the restore drill manually with:

./jeannie drill-gitea-restore
./jeannie drill-restore
./jeannie drill-pihole-restore

drill-restore validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes.

Useful checks:

./jeannie backup-status
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Heimdall

Heimdall is external bootstrap infrastructure. It runs on the Debian host as an always-on Docker Compose service from infra/heimdall/docker-compose.yml, not as a Kubernetes workload. This keeps the lab navigation page available when the cluster is destroyed, rebuilding, or unhealthy.

Deploy or refresh it from the Debian server with:

./jeannie deploy-heimdall

./jeannie up also runs this step during the independent host services stage unless LAB_HEIMDALL_DEPLOY=false is set. The default data path is /data/homelab-heimdall/data/config, and the local HTTP port is 8082. Dashboard links are seeded from infra/heimdall/links.json by a sidecar container that upserts repo-managed tiles without deleting manually added ones.

The Debian bare repo remains the GitOps mirror:

/home/jv/git-server/my-homelab-configs.git

Argo CD consumes that Debian mirror through the default gitops_repo_url. Gitea Actions pushes the main commit into the mirror before running the selected deploy command.

The platform bootstrap registers the Argo CD repository secret and the SSH host key for the Debian GitOps mirror. If Argo CD reports knownhosts: key is unknown after the Debian host was rebuilt or its SSH host key changed, refresh argocd-ssh-known-hosts-cm in the argocd namespace, restart argocd-repo-server, and hard-refresh the affected Application.

Deploy or refresh the external Gitea container from the Debian host with:

./jeannie deploy-gitea

Gitea Backups

./jeannie up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the configured Gitea host, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Gitea data directory.

Run a manual backup from the Debian server with:

./jeannie backup-gitea

Run the restore drill manually with:

./jeannie drill-gitea-restore
./jeannie drill-restore
./jeannie drill-pihole-restore

drill-restore validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes.

Useful checks:

./jeannie backup-status
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Gitea Backups

./jeannie up installs a Debian-host systemd timer named homelab-gitea-backup.timer. The timer runs daily, SSHes to the configured Gitea host, executes gitea dump inside the Gitea Docker container, copies the dump back to Debian, and stores it under /home/jv/backups/gitea. The default retention is 30 days.

The same install step also creates homelab-gitea-restore-drill.timer. The monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it to a temporary directory, records a report under /home/jv/backups/gitea-restore-drills, and removes the temporary extract. It does not write into the live Gitea data directory.

Run a manual backup from the Debian server with:

./jeannie backup-gitea

Run the restore drill manually with:

./jeannie drill-gitea-restore
./jeannie drill-restore
./jeannie drill-pihole-restore

drill-restore validates the latest Gitea backup archive, Pi-hole repo-managed config inputs, the latest OpenTofu state backup archive when present, and the GitOps rebuild path. It does not replace running services or mutate Kubernetes.

Useful checks:

./jeannie backup-status
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Gitea Actions

This repo includes a Gitea Actions workflow at .gitea/workflows/homelab-main.yml. It runs validation on pushes to dev and main, and deploys only from main. That keeps the promotion path simple:

dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync

The workflow targets a repository-scoped Debian host runner with the label homelab-debian.

Run the same validation locally before promoting:

./jeannie validate

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Identity and access audit

Run a read-only access inventory from the Debian server with:

./jeannie access-audit

The audit reports configured SSH targets and key file modes, Git/Gitea remotes, Tailscale ACL policy validation, current Kubernetes authorization, broad cluster-admin bindings, automation service accounts, and Gitea runner status. It does not create users, tokens, kubeconfigs, or keys. Use it to move toward separate identities:

• admin kubeconfig only on the Debian control host
• read-only kubeconfig for dashboards and audits
• separate service accounts for automation
• explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
• repo-managed Tailscale ACLs

The read-only Kubernetes identity is managed in apps/access-control. After that app syncs, generate a separate read-only kubeconfig on the Debian host:

./jeannie kubeconfig-readonly

The workflow only blocks automatic deploy for external Gitea service changes: files under infra/gitea/, or edits inside the deploy_gitea, install_gitea_backup_timer, backup_gitea, or drill_gitea_restore functions in jeannie. Other changes use HOMELAB_ACTION_COMMAND=auto by default: Actions runs ./jeannie doctor-versions when the Debian runner already has a cluster kubeconfig. If node versions are aligned it runs ./jeannie apps; if kubelet minor drift is detected it runs ./jeannie rebuild-cluster; if no kubeconfig exists it also runs ./jeannie rebuild-cluster.

Set HOMELAB_ACTION_COMMAND=apps or HOMELAB_ACTION_COMMAND=rebuild-cluster on the runner to force one path.

./jeannie bootstrap-gitea-repo also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's gitea remote to ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git. The default key is /home/jv/.ssh/id_ed25519.pub; set LAB_GITEA_REPO_SSH_KEY_PATH to use a different Debian-host key, or LAB_GITEA_REPO_SSH_BOOTSTRAP=false to leave SSH access unchanged. The Actions deploy job uses the checked-out Actions workspace as the source commit, updates the first available persistent checkout from HOMELAB_DEPLOY_DIR, /home/jv/my-homelab-configs, or /home/jv/repos/my-homelab-configs, and otherwise deploys directly from the Actions workspace. It does not need SSH read access back to Gitea.

Enable Actions for the repository in Gitea, then create a repository-level runner token from:

https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners

Register and start the Debian runner from the Debian server:

cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner

The runner is installed as homelab-gitea-runner.service, runs as user jv, and uses a host label instead of a Docker job container because deployment needs the Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.

The deployment job is non-interactive. User jv must be able to run sudo -n true on the Debian host for deployment commands that require sudo.

Useful checks:

systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Renovate

renovate.json defines dependency update rules for Dockerfiles, OpenTofu providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the normal Gitea Actions path, and run ./jeannie up manually on the Debian server for platform or provisioning updates.

Destructive Rebuilds

./jeannie nuke resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under /data/openebs/local.

For multi-node labs, set WORKER_SSH_TARGETS to a space-separated list of SSH targets. It defaults to an empty string so worker nodes are not cleaned unless you explicitly include them.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.

Website App

The website is a PHP app under apps/website. It includes a home page, CV page, blog page, and demos page, plus a lightweight translation flow backed by Redis, n8n, and Ollama. Static language files live in apps/website/lang; en.php and nah.php are curated source files, with the Nahuatl home page intentionally biased toward as many Nahuatl words as possible while keeping technical terms understandable. Unsupported browser languages use the same-origin /translate.php endpoint, which calls Ollama server-side through OLLAMA_HOST and OLLAMA_MODEL; the browser never calls the private Ollama IP directly. Redis caches per-string translation results in the website-production namespace, while n8n is available for translation prewarming and batch workflow jobs. The default model is the custom website-translator Ollama model defined in apps/website/ollama/Modelfile. Generated runtime language JSON is saved through save_lang.php on the website PVC, and translate.php emits structured logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.

Create or refresh the Ollama model on the Debian server before deploying a website image that points at it:

./jeannie website-translation-model

Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach OLLAMA_HOST=http://192.168.100.73:11434:

./jeannie website-ollama-listen

The Debian host bootstrap also manages Ollama when debian_pc_install_ollama is true: it installs the service, stores models under /data/ollama/models, binds the API to the configured LAN listener, and pulls the lightweight ai_gateway.model. On an existing host, apply just this setup with:

./jeannie ollama-setup

jeannie also has a backstage local AI helper for doctor commands. It is not a separate CLI action; when ai_gateway.enabled is true in homelab.yml, doctor commands try the configured Ollama model and silently skip the helper if Ollama is down. The default lightweight model is:

ollama pull qwen2.5:0.5b

Disable it for low-CPU sessions with:

LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge

Build the local homelab knowledge index separately from the main deployment:

./jeannie ai-index
./jeannie ai-check
./jeannie ask "how do I check edge to Gitea?"

The index is built from the non-secret source list in infra/ai/knowledge-sources.txt, skips paths in infra/ai/knowledge-excludes.txt, and defaults to /data/homelab-ai/index. Doctor commands use it as extra context when it exists, but infrastructure deployment does not depend on it. Use ask as the runbook and command finder. It returns the closest indexed docs/scripts by default; set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.

The CV page has two client-side presentation modes:

• Elegant: dark, minimal, terminal-inspired styling with a square profile image and light green console text.
• Fancy: centered circular profile image, cursive orbit text, and a cursor-following portrait rotation effect.

The Demos page is a catalog in the PHP website. The actual demo applications are served from a separate demos-static artifact under apps/demos-static and are published through the demos-static Argo CD application. Public traffic reaches them through the edge path at /demo-apps/.

./jeannie up builds and pushes two independent images:

• a content-hash php-website tag generated by jeannie and passed to Argo CD as a Kustomize image override
• demos-static:latest from apps/demos-static
• Cosign signatures and signed SPDX SBOM attestations for both pushed images

The website manifest keeps the stable base image name php-website:bootstrap. During bootstrap, jeannie hashes apps/website, builds <registry>/php-website:src-<hash>, exports that exact reference through TF_VAR_website_image_ref, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested.

After ./jeannie apps, the live deployment image should be a content-hash tag, for example 192.168.100.73:30500/php-website:src-.... If it still shows php-website:latest, Argo CD has not rendered the current Application source. Check the website-production Application source, sync status, and repository access before restarting pods.

The first demo, The Client-Side Media Cruncher (Wasm + TS), currently performs private, browser-only image compression and conversion using native Canvas APIs. Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled to WebAssembly with a TypeScript UI so the codec work stays fast and still avoids backend uploads.

The demos are designed to be local-first so the current cluster can serve them from any Linux app node without turning either pod into an application server. The website pod serves the portfolio shell and the demos-static pod serves static demo bundles; CPU-heavy work runs in the visitor's browser. Because the deployments can run on either Debian or ARM workers, avoid bundling large ML models, server-side WebSocket probes, or backend video transcoders into either image. If those demos become production-grade, lazy load model assets in the browser or move backend workers to a larger node, such as VMs on the Orange Pi 5 Plus.

Current demo inventory:

• Client-side media cruncher: image conversion/compression with Canvas; future Rust/Wasm codec path for video.
• Internet quality visualizer: live Canvas graph for latency, jitter, and stability using same-origin browser probes; a dedicated WebSocket echo endpoint would be the production version.
• Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and local text-log filtering.
• Architecture simulator: click-driven load, crash, and auto-scale simulation.
• Offline traveler converter: PWA shell with timezone, currency, and GB/GiB conversions.
• Privacy-first redactor: local image redaction prototype; future onnxruntime-web plus quantized YOLO or face model path.
• Local sentiment sandbox: lightweight local sentiment, keyword, and summary prototype; future Transformers.js/ONNX path.
• Model drift simulator: visual MLOps playground for spikes, corrupted inputs, and retraining.

The Kubernetes deployment uses apps/website/web-app.yaml as a Kustomize base. Keep TF_VAR_registry_endpoint aligned with the local registry endpoint used by the app image build and with the image globs in apps/supply-chain-policy/local-registry-image-policy.yaml.

Keep the .terraform.lock.hcl files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds.