2625 lines
110 KiB
Markdown
2625 lines
110 KiB
Markdown
# Application Operations
|
|
|
|
## Adding Apps
|
|
|
|
Add Kubernetes manifests under `apps/<name>` and register them in
|
|
`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and
|
|
self-healing for the app.
|
|
|
|
The `heimdall` Argo CD app name is retained as a small route bundle for
|
|
Grafana, Prometheus, Alertmanager, and Argo CD ingresses so the next app sync can
|
|
prune the old in-cluster Heimdall workload cleanly. Do not add new Kubernetes
|
|
workloads there. Use `./jeannie deploy-heimdall` for the dashboard
|
|
itself.
|
|
|
|
The `n8n` app runs in the `n8n` namespace with retained OpenEBS storage and is
|
|
exposed at `https://n8n.lab2025.duckdns.org`. It receives
|
|
`OLLAMA_BASE_URL=http://192.168.100.73:11434` so workflows can call the existing
|
|
Debian-host Ollama API for translation prewarming and batch jobs.
|
|
|
|
The ARR/media stack is intentionally not deployed by the Kubernetes app
|
|
pipeline. It is better suited to a Docker Compose stack on the Debian host with
|
|
persistent volumes under `/data`, where large media paths, qBittorrent writes,
|
|
and service config directories can survive cluster rebuilds without OpenEBS
|
|
node-affinity or PVC migration concerns. The active host-level Compose
|
|
definition lives under `infra/arr-stack` and stores persistent data under
|
|
`/data/arr`. Historical Kubernetes manifests are archived under
|
|
`archive/apps/arr-stack` for migration reference only.
|
|
|
|
## Storage
|
|
|
|
OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use
|
|
retained local PV paths such as `/data/openebs/local/registry`; these paths are
|
|
intentionally outside kubeadm reset paths so data can survive cluster
|
|
destroy/create cycles. Those critical volumes are declared explicitly as
|
|
retained local PVs so a rebuilt cluster binds back to the same host paths
|
|
instead of creating fresh directories.
|
|
|
|
For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
|
|
standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
|
|
data still lives under `/data/openebs/local`, and larger service data such as
|
|
Gitea, Ollama models, and app volumes should stay under `/data`.
|
|
|
|
## Gitea
|
|
|
|
Gitea is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
|
|
Kubernetes workload. This keeps Git available when the Kubernetes cluster is
|
|
destroyed and rebuilt.
|
|
|
|
The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
|
|
Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.
|
|
|
|
Public source browsing stays available through
|
|
`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
|
|
can view public repositories, so the blog can link to code read-only while
|
|
writes still require an authenticated Gitea account.
|
|
|
|
## Heimdall
|
|
|
|
Heimdall is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not
|
|
as a Kubernetes workload. This keeps the lab navigation page available when the
|
|
cluster is destroyed, rebuilding, or unhealthy.
|
|
|
|
Deploy or refresh it from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie deploy-heimdall
|
|
```
|
|
|
|
`./jeannie up` also runs this step during the independent host
|
|
services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path
|
|
is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`.
|
|
Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar
|
|
container that upserts repo-managed tiles without deleting manually added ones.
|
|
|
|
The Debian bare repo remains the GitOps mirror:
|
|
|
|
```text
|
|
/home/jv/git-server/my-homelab-configs.git
|
|
```
|
|
|
|
Argo CD consumes that Debian mirror through the default `gitops_repo_url`.
|
|
Gitea Actions pushes the `main` commit into the mirror before running the
|
|
selected deploy command.
|
|
|
|
The platform bootstrap registers the Argo CD repository secret and the SSH host
|
|
key for the Debian GitOps mirror. If Argo CD reports
|
|
`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host
|
|
key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace,
|
|
restart `argocd-repo-server`, and hard-refresh the affected Application.
|
|
|
|
Deploy or refresh the external Gitea container from the Debian host with:
|
|
|
|
```bash
|
|
./jeannie deploy-gitea
|
|
```
|
|
|
|
## Gitea Backups
|
|
|
|
`./jeannie up` installs a Debian-host systemd timer named
|
|
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
|
|
Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
|
|
dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
|
|
retention is 30 days.
|
|
|
|
The same install step also creates `homelab-gitea-restore-drill.timer`. The
|
|
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
|
|
to a temporary directory, records a report under
|
|
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
|
|
does not write into the live Gitea data directory.
|
|
|
|
Run a manual backup from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie backup-gitea
|
|
```
|
|
|
|
Run the restore drill manually with:
|
|
|
|
```bash
|
|
./jeannie drill-gitea-restore
|
|
./jeannie drill-restore
|
|
./jeannie drill-pihole-restore
|
|
```
|
|
|
|
`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
|
|
config inputs, the latest OpenTofu state backup archive when present, and the
|
|
GitOps rebuild path. It does not replace running services or mutate Kubernetes.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
./jeannie backup-status
|
|
systemctl list-timers homelab-gitea-backup.timer
|
|
systemctl list-timers homelab-gitea-restore-drill.timer
|
|
sudo systemctl start homelab-gitea-backup.service
|
|
ls -lh /home/jv/backups/gitea
|
|
ls -lh /home/jv/backups/gitea-restore-drills
|
|
```
|
|
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Storage
|
|
|
|
OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use
|
|
retained local PV paths such as `/data/openebs/local/registry`; these paths are
|
|
intentionally outside kubeadm reset paths so data can survive cluster
|
|
destroy/create cycles. Those critical volumes are declared explicitly as
|
|
retained local PVs so a rebuilt cluster binds back to the same host paths
|
|
instead of creating fresh directories.
|
|
|
|
For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
|
|
standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
|
|
data still lives under `/data/openebs/local`, and larger service data such as
|
|
Gitea, Ollama models, and app volumes should stay under `/data`.
|
|
|
|
## Gitea
|
|
|
|
Gitea is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
|
|
Kubernetes workload. This keeps Git available when the Kubernetes cluster is
|
|
destroyed and rebuilt.
|
|
|
|
The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
|
|
Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.
|
|
|
|
Public source browsing stays available through
|
|
`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
|
|
can view public repositories, so the blog can link to code read-only while
|
|
writes still require an authenticated Gitea account.
|
|
|
|
## Heimdall
|
|
|
|
Heimdall is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not
|
|
as a Kubernetes workload. This keeps the lab navigation page available when the
|
|
cluster is destroyed, rebuilding, or unhealthy.
|
|
|
|
Deploy or refresh it from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie deploy-heimdall
|
|
```
|
|
|
|
`./jeannie up` also runs this step during the independent host
|
|
services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path
|
|
is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`.
|
|
Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar
|
|
container that upserts repo-managed tiles without deleting manually added ones.
|
|
|
|
The Debian bare repo remains the GitOps mirror:
|
|
|
|
```text
|
|
/home/jv/git-server/my-homelab-configs.git
|
|
```
|
|
|
|
Argo CD consumes that Debian mirror through the default `gitops_repo_url`.
|
|
Gitea Actions pushes the `main` commit into the mirror before running the
|
|
selected deploy command.
|
|
|
|
The platform bootstrap registers the Argo CD repository secret and the SSH host
|
|
key for the Debian GitOps mirror. If Argo CD reports
|
|
`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host
|
|
key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace,
|
|
restart `argocd-repo-server`, and hard-refresh the affected Application.
|
|
|
|
Deploy or refresh the external Gitea container from the Debian host with:
|
|
|
|
```bash
|
|
./jeannie deploy-gitea
|
|
```
|
|
|
|
## Gitea Backups
|
|
|
|
`./jeannie up` installs a Debian-host systemd timer named
|
|
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
|
|
Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
|
|
dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
|
|
retention is 30 days.
|
|
|
|
The same install step also creates `homelab-gitea-restore-drill.timer`. The
|
|
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
|
|
to a temporary directory, records a report under
|
|
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
|
|
does not write into the live Gitea data directory.
|
|
|
|
Run a manual backup from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie backup-gitea
|
|
```
|
|
|
|
Run the restore drill manually with:
|
|
|
|
```bash
|
|
./jeannie drill-gitea-restore
|
|
./jeannie drill-restore
|
|
./jeannie drill-pihole-restore
|
|
```
|
|
|
|
`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
|
|
config inputs, the latest OpenTofu state backup archive when present, and the
|
|
GitOps rebuild path. It does not replace running services or mutate Kubernetes.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
./jeannie backup-status
|
|
systemctl list-timers homelab-gitea-backup.timer
|
|
systemctl list-timers homelab-gitea-restore-drill.timer
|
|
sudo systemctl start homelab-gitea-backup.service
|
|
ls -lh /home/jv/backups/gitea
|
|
ls -lh /home/jv/backups/gitea-restore-drills
|
|
```
|
|
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Gitea
|
|
|
|
Gitea is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
|
|
Kubernetes workload. This keeps Git available when the Kubernetes cluster is
|
|
destroyed and rebuilt.
|
|
|
|
The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
|
|
Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.
|
|
|
|
Public source browsing stays available through
|
|
`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
|
|
can view public repositories, so the blog can link to code read-only while
|
|
writes still require an authenticated Gitea account.
|
|
|
|
## Heimdall
|
|
|
|
Heimdall is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not
|
|
as a Kubernetes workload. This keeps the lab navigation page available when the
|
|
cluster is destroyed, rebuilding, or unhealthy.
|
|
|
|
Deploy or refresh it from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie deploy-heimdall
|
|
```
|
|
|
|
`./jeannie up` also runs this step during the independent host
|
|
services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path
|
|
is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`.
|
|
Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar
|
|
container that upserts repo-managed tiles without deleting manually added ones.
|
|
|
|
The Debian bare repo remains the GitOps mirror:
|
|
|
|
```text
|
|
/home/jv/git-server/my-homelab-configs.git
|
|
```
|
|
|
|
Argo CD consumes that Debian mirror through the default `gitops_repo_url`.
|
|
Gitea Actions pushes the `main` commit into the mirror before running the
|
|
selected deploy command.
|
|
|
|
The platform bootstrap registers the Argo CD repository secret and the SSH host
|
|
key for the Debian GitOps mirror. If Argo CD reports
|
|
`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host
|
|
key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace,
|
|
restart `argocd-repo-server`, and hard-refresh the affected Application.
|
|
|
|
Deploy or refresh the external Gitea container from the Debian host with:
|
|
|
|
```bash
|
|
./jeannie deploy-gitea
|
|
```
|
|
|
|
## Gitea Backups
|
|
|
|
`./jeannie up` installs a Debian-host systemd timer named
|
|
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
|
|
Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
|
|
dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
|
|
retention is 30 days.
|
|
|
|
The same install step also creates `homelab-gitea-restore-drill.timer`. The
|
|
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
|
|
to a temporary directory, records a report under
|
|
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
|
|
does not write into the live Gitea data directory.
|
|
|
|
Run a manual backup from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie backup-gitea
|
|
```
|
|
|
|
Run the restore drill manually with:
|
|
|
|
```bash
|
|
./jeannie drill-gitea-restore
|
|
./jeannie drill-restore
|
|
./jeannie drill-pihole-restore
|
|
```
|
|
|
|
`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
|
|
config inputs, the latest OpenTofu state backup archive when present, and the
|
|
GitOps rebuild path. It does not replace running services or mutate Kubernetes.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
./jeannie backup-status
|
|
systemctl list-timers homelab-gitea-backup.timer
|
|
systemctl list-timers homelab-gitea-restore-drill.timer
|
|
sudo systemctl start homelab-gitea-backup.service
|
|
ls -lh /home/jv/backups/gitea
|
|
ls -lh /home/jv/backups/gitea-restore-drills
|
|
```
|
|
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Heimdall
|
|
|
|
Heimdall is external bootstrap infrastructure. It runs on the Debian host as an
|
|
always-on Docker Compose service from `infra/heimdall/docker-compose.yml`, not
|
|
as a Kubernetes workload. This keeps the lab navigation page available when the
|
|
cluster is destroyed, rebuilding, or unhealthy.
|
|
|
|
Deploy or refresh it from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie deploy-heimdall
|
|
```
|
|
|
|
`./jeannie up` also runs this step during the independent host
|
|
services stage unless `LAB_HEIMDALL_DEPLOY=false` is set. The default data path
|
|
is `/data/homelab-heimdall/data/config`, and the local HTTP port is `8082`.
|
|
Dashboard links are seeded from `infra/heimdall/links.json` by a sidecar
|
|
container that upserts repo-managed tiles without deleting manually added ones.
|
|
|
|
The Debian bare repo remains the GitOps mirror:
|
|
|
|
```text
|
|
/home/jv/git-server/my-homelab-configs.git
|
|
```
|
|
|
|
Argo CD consumes that Debian mirror through the default `gitops_repo_url`.
|
|
Gitea Actions pushes the `main` commit into the mirror before running the
|
|
selected deploy command.
|
|
|
|
The platform bootstrap registers the Argo CD repository secret and the SSH host
|
|
key for the Debian GitOps mirror. If Argo CD reports
|
|
`knownhosts: key is unknown` after the Debian host was rebuilt or its SSH host
|
|
key changed, refresh `argocd-ssh-known-hosts-cm` in the `argocd` namespace,
|
|
restart `argocd-repo-server`, and hard-refresh the affected Application.
|
|
|
|
Deploy or refresh the external Gitea container from the Debian host with:
|
|
|
|
```bash
|
|
./jeannie deploy-gitea
|
|
```
|
|
|
|
## Gitea Backups
|
|
|
|
`./jeannie up` installs a Debian-host systemd timer named
|
|
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
|
|
Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
|
|
dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
|
|
retention is 30 days.
|
|
|
|
The same install step also creates `homelab-gitea-restore-drill.timer`. The
|
|
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
|
|
to a temporary directory, records a report under
|
|
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
|
|
does not write into the live Gitea data directory.
|
|
|
|
Run a manual backup from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie backup-gitea
|
|
```
|
|
|
|
Run the restore drill manually with:
|
|
|
|
```bash
|
|
./jeannie drill-gitea-restore
|
|
./jeannie drill-restore
|
|
./jeannie drill-pihole-restore
|
|
```
|
|
|
|
`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
|
|
config inputs, the latest OpenTofu state backup archive when present, and the
|
|
GitOps rebuild path. It does not replace running services or mutate Kubernetes.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
./jeannie backup-status
|
|
systemctl list-timers homelab-gitea-backup.timer
|
|
systemctl list-timers homelab-gitea-restore-drill.timer
|
|
sudo systemctl start homelab-gitea-backup.service
|
|
ls -lh /home/jv/backups/gitea
|
|
ls -lh /home/jv/backups/gitea-restore-drills
|
|
```
|
|
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Gitea Backups
|
|
|
|
`./jeannie up` installs a Debian-host systemd timer named
|
|
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
|
|
Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
|
|
dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
|
|
retention is 30 days.
|
|
|
|
The same install step also creates `homelab-gitea-restore-drill.timer`. The
|
|
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
|
|
to a temporary directory, records a report under
|
|
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
|
|
does not write into the live Gitea data directory.
|
|
|
|
Run a manual backup from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie backup-gitea
|
|
```
|
|
|
|
Run the restore drill manually with:
|
|
|
|
```bash
|
|
./jeannie drill-gitea-restore
|
|
./jeannie drill-restore
|
|
./jeannie drill-pihole-restore
|
|
```
|
|
|
|
`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
|
|
config inputs, the latest OpenTofu state backup archive when present, and the
|
|
GitOps rebuild path. It does not replace running services or mutate Kubernetes.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
./jeannie backup-status
|
|
systemctl list-timers homelab-gitea-backup.timer
|
|
systemctl list-timers homelab-gitea-restore-drill.timer
|
|
sudo systemctl start homelab-gitea-backup.service
|
|
ls -lh /home/jv/backups/gitea
|
|
ls -lh /home/jv/backups/gitea-restore-drills
|
|
```
|
|
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Gitea Actions
|
|
|
|
This repo includes a Gitea Actions workflow at
|
|
`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
|
|
`main`, and deploys only from `main`. That keeps the promotion path simple:
|
|
|
|
```text
|
|
dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
|
|
```
|
|
|
|
The workflow targets a repository-scoped Debian host runner with the label
|
|
`homelab-debian`.
|
|
|
|
Run the same validation locally before promoting:
|
|
|
|
```bash
|
|
./jeannie validate
|
|
```
|
|
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Identity and access audit
|
|
|
|
Run a read-only access inventory from the Debian server with:
|
|
|
|
```bash
|
|
./jeannie access-audit
|
|
```
|
|
|
|
The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
|
|
Tailscale ACL policy validation, current Kubernetes authorization, broad
|
|
cluster-admin bindings, automation service accounts, and Gitea runner status.
|
|
It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
|
|
separate identities:
|
|
|
|
- admin kubeconfig only on the Debian control host
|
|
- read-only kubeconfig for dashboards and audits
|
|
- separate service accounts for automation
|
|
- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
|
|
- repo-managed Tailscale ACLs
|
|
|
|
The read-only Kubernetes identity is managed in `apps/access-control`. After
|
|
that app syncs, generate a separate read-only kubeconfig on the Debian host:
|
|
|
|
```bash
|
|
./jeannie kubeconfig-readonly
|
|
```
|
|
|
|
The workflow only blocks automatic deploy for external Gitea service
|
|
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
|
|
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
|
|
functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
|
|
default: Actions runs `./jeannie doctor-versions` when the Debian runner already
|
|
has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
|
|
if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
|
|
kubeconfig exists it also runs `./jeannie rebuild-cluster`.
|
|
|
|
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
|
|
on the runner to force one path.
|
|
|
|
`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
|
|
with the Gitea repository and switches the Debian working copy's `gitea` remote
|
|
to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
|
|
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
|
|
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
|
|
access unchanged. The Actions deploy job uses the checked-out Actions workspace
|
|
as the source commit, updates the first available persistent checkout from
|
|
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
|
|
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
|
|
Actions workspace. It does not need SSH read access back to Gitea.
|
|
|
|
Enable Actions for the repository in Gitea, then create a repository-level runner
|
|
token from:
|
|
|
|
```text
|
|
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
|
|
```
|
|
|
|
Register and start the Debian runner from the Debian server:
|
|
|
|
```bash
|
|
cd ~/my-homelab-configs
|
|
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
|
|
```
|
|
|
|
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
|
|
uses a host label instead of a Docker job container because deployment needs the
|
|
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
|
|
|
|
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
|
|
true` on the Debian host for deployment commands that require sudo.
|
|
|
|
Useful checks:
|
|
|
|
```bash
|
|
systemctl status homelab-gitea-runner.service
|
|
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
|
|
```
|
|
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Renovate
|
|
|
|
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
|
|
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
|
|
workflow. Renovate should open reviewable update branches or PRs only; it must
|
|
not auto-merge infrastructure changes. Keep app-only dependency updates on the
|
|
normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
|
|
for platform or provisioning updates.
|
|
|
|
## Destructive Rebuilds
|
|
|
|
`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
|
|
links, iptables rules, and local OpenTofu state. It does not delete retained data
|
|
under `/data/openebs/local`.
|
|
|
|
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
|
|
targets. It defaults to an empty string so worker nodes are not cleaned unless
|
|
you explicitly include them.
|
|
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|
|
## Website App
|
|
|
|
The website is a PHP app under `apps/website`. It includes a home page, CV page,
|
|
blog page, and demos page, plus a lightweight translation flow backed by Redis,
|
|
n8n, and Ollama.
|
|
Static language files live in `apps/website/lang`; `en.php` and `nah.php` are
|
|
curated source files, with the Nahuatl home page intentionally biased toward as
|
|
many Nahuatl words as possible while keeping technical terms understandable.
|
|
Unsupported browser languages use the same-origin `/translate.php` endpoint,
|
|
which calls Ollama server-side through `OLLAMA_HOST` and `OLLAMA_MODEL`; the
|
|
browser never calls the private Ollama IP directly. Redis caches per-string
|
|
translation results in the `website-production` namespace, while n8n is
|
|
available for translation prewarming and batch workflow jobs. The default model
|
|
is the custom `website-translator` Ollama model defined in
|
|
`apps/website/ollama/Modelfile`. Generated runtime language JSON is saved
|
|
through `save_lang.php` on the website PVC, and `translate.php` emits structured
|
|
logs for cache hits, misses, Ollama latency, JSON parse failures, and timeouts.
|
|
|
|
Create or refresh the Ollama model on the Debian server before deploying a
|
|
website image that points at it:
|
|
|
|
```bash
|
|
./jeannie website-translation-model
|
|
```
|
|
|
|
Ollama must also listen on the Debian host LAN address so Kubernetes pods on
|
|
other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
|
|
|
|
```bash
|
|
./jeannie website-ollama-listen
|
|
```
|
|
|
|
The Debian host bootstrap also manages Ollama when `debian_pc_install_ollama`
|
|
is true: it installs the service, stores models under `/data/ollama/models`,
|
|
binds the API to the configured LAN listener, and pulls the lightweight
|
|
`ai_gateway.model`. On an existing host, apply just this setup with:
|
|
|
|
```bash
|
|
./jeannie ollama-setup
|
|
```
|
|
|
|
`jeannie` also has a backstage local AI helper for doctor commands.
|
|
It is not a separate CLI action; when `ai_gateway.enabled` is true in
|
|
`homelab.yml`, doctor commands try the configured Ollama model and silently
|
|
skip the helper if Ollama is down. The default lightweight model is:
|
|
|
|
```bash
|
|
ollama pull qwen2.5:0.5b
|
|
```
|
|
|
|
Disable it for low-CPU sessions with:
|
|
|
|
```bash
|
|
LAB_BACKSTAGE_BRAIN_ENABLED=false ./jeannie doctor-edge
|
|
```
|
|
|
|
Build the local homelab knowledge index separately from the main deployment:
|
|
|
|
```bash
|
|
./jeannie ai-index
|
|
./jeannie ai-check
|
|
./jeannie ask "how do I check edge to Gitea?"
|
|
```
|
|
|
|
The index is built from the non-secret source list in
|
|
`infra/ai/knowledge-sources.txt`, skips paths in
|
|
`infra/ai/knowledge-excludes.txt`, and defaults to `/data/homelab-ai/index`.
|
|
Doctor commands use it as extra context when it exists, but infrastructure
|
|
deployment does not depend on it.
|
|
Use `ask` as the runbook and command finder. It returns the closest indexed
|
|
docs/scripts by default; set `LAB_AI_ASK_LLM=true` to ask Ollama after
|
|
retrieval.
|
|
|
|
The CV page has two client-side presentation modes:
|
|
|
|
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
|
|
image and light green console text.
|
|
- `Fancy`: centered circular profile image, cursive orbit text, and a
|
|
cursor-following portrait rotation effect.
|
|
|
|
The Demos page is a catalog in the PHP website. The actual demo applications are
|
|
served from a separate `demos-static` artifact under `apps/demos-static` and are
|
|
published through the `demos-static` Argo CD application. Public traffic reaches
|
|
them through the edge path at `/demo-apps/`.
|
|
|
|
`./jeannie up` builds and pushes two independent images:
|
|
|
|
- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
|
|
as a Kustomize image override
|
|
- `demos-static:latest` from `apps/demos-static`
|
|
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
|
|
|
|
The website manifest keeps the stable base image name `php-website:bootstrap`.
|
|
During bootstrap, `jeannie` hashes `apps/website`, builds
|
|
`<registry>/php-website:src-<hash>`, exports that exact reference through
|
|
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through
|
|
Kustomize. This keeps the GitOps source generic while the deployed image remains
|
|
immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
|
|
image digest, so the workload runs the same digest that was signed and attested.
|
|
|
|
After `./jeannie apps`, the live deployment image should be a content-hash tag,
|
|
for example `192.168.100.73:30500/php-website:src-...`. If it still shows
|
|
`php-website:latest`, Argo CD has not rendered the current Application source.
|
|
Check the `website-production` Application source, sync status, and repository
|
|
access before restarting pods.
|
|
|
|
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
|
|
private, browser-only image compression and conversion using native Canvas APIs.
|
|
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
|
|
to WebAssembly with a TypeScript UI so the codec work stays fast and still
|
|
avoids backend uploads.
|
|
|
|
The demos are designed to be local-first so the current cluster can serve them
|
|
from any Linux app node without turning either pod into an application server.
|
|
The website pod serves the portfolio shell and the `demos-static` pod serves
|
|
static demo bundles; CPU-heavy work runs in the visitor's browser. Because the
|
|
deployments can run on either Debian or ARM workers, avoid bundling large ML
|
|
models, server-side WebSocket probes, or backend video transcoders into either
|
|
image. If those demos become production-grade, lazy load model assets in the
|
|
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
|
|
Plus.
|
|
|
|
Current demo inventory:
|
|
|
|
- Client-side media cruncher: image conversion/compression with Canvas; future
|
|
Rust/Wasm codec path for video.
|
|
- Internet quality visualizer: live Canvas graph for latency, jitter, and
|
|
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
|
|
would be the production version.
|
|
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
|
|
local text-log filtering.
|
|
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
|
|
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
|
|
conversions.
|
|
- Privacy-first redactor: local image redaction prototype; future
|
|
onnxruntime-web plus quantized YOLO or face model path.
|
|
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
|
|
prototype; future Transformers.js/ONNX path.
|
|
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
|
|
and retraining.
|
|
|
|
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
|
|
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
|
|
the app image build and with the image globs in
|
|
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
|
|
|
|
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
|
|
make bootstrap behavior reproducible across nodes and rebuilds.
|