# JEANNIE(1) ## NAME jeannie - operate the personal homelab control pipeline ## SYNOPSIS ```sh ./jeannie COMMAND [ARG...] ``` ## DESCRIPTION `jeannie` is the main homelab operator script. It deploys and checks the Debian control host, Gitea, Raspberry Pi services, Pimox workers, Kubernetes platform components, applications, public edge routing, secrets tooling, and local AI helpers. Run mutating infrastructure commands from the Debian homelab host. Commands that can destroy or rewrite state have explicit confirmation environment variables. The canonical script name is `metadata.main_script` in `homelab.yml`. ## COMMANDS ### Core Lifecycle `up` : Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi services, runs full preflight and pre-apply doctor checks, then applies Pimox, OpenWrt, cluster, platform, apps, and edge stacks. `plan [all|provisioning|cluster|platform|apps|edge]` : Run OpenTofu plans without applying changes. The default target is `all`. `rebuild-cluster` : Back up current local OpenTofu state, destroy cluster state through the guarded nuke path, rebuild Pimox workers, recreate the cluster, and redeploy platform, apps, and edge. `stop-cluster` : Stop the Kubernetes control-plane runtime and worker VMs without destroying cluster state. `start-cluster` : Start the Kubernetes control-plane runtime and desired worker VMs. `status` : Print a cascade health report for the configured homelab, from bootstrap readiness through public website health. Set `LAB_STATUS_DETAILS=true` to append the older detailed tables. The cascade includes "what broke" signals for recent deployment readiness, pod restarts, node pressure, disk pressure, Traefik 5xx/404 log evidence, and recent Gitea errors. Prometheus alerts : High-signal alert rules are managed as the `apps/homelab-alerts` GitOps app. They cover node readiness, restart storms, unavailable deployments, storage pressure, Traefik 5xx, and core service visibility. `validate` : Run non-mutating repository checks used by the GitOps promotion flow: inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu formatting, tailnet policy validation, and optional static security checks. `access-audit` : Run a read-only access inventory across SSH targets, Git/Gitea remotes, Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the Gitea runner. `nuke` : Destroy Kubernetes state and Pimox worker VMs. Requires `LAB_CONFIRM_NUKE=homelab`. ### Deploy Targets `apps` : Deploy platform applications and build or publish application images when needed. `deploy-gitea` : Deploy the external Gitea Docker Compose service on the Debian host. `rpi-services` : Deploy Pi-hole, Unbound, and Uptime Kuma on the Raspberry Pi. `bootstrap-gitea-repo` : Ensure the configured Gitea repository, branch, remotes, and host SSH key are ready. `openwrt` : Apply the OpenWrt VM stack. ### Gitea Operations `backup-gitea` : Run a Gitea backup on the Debian host. `drill-gitea-restore` : Run the Gitea restore drill workflow. `install-gitea-runner [TOKEN]` : Install or repair the Gitea Actions runner. If `TOKEN` is omitted, the command uses the configured token source. ### Checks And Doctors `preflight` : Run homelab preflight checks. This verifies inventory-derived host settings, Docker roots, Tailscale IPs, Gitea reachability, Pimox storage, and OCI edge SSH. `doctor-preapply` : Run blocking readiness checks used by `up` before high-blast-radius applies. It checks Debian free disk, Docker and containerd, Raspberry Pi Docker writability, Pimox storage, OCI edge disk, and Pi-hole DNS. `doctor-versions` : Check Kubernetes component version alignment. `doctor-edge` : Diagnose public edge routing, Traefik reachability, and Gitea edge exposure. `doctor-gitea` : Diagnose Debian Gitea container, local access, edge access, and repository health. `doctor-rpi` : Diagnose Raspberry Pi services, Pi-hole DNS, Unbound DNS, fallback DNS, Uptime Kuma, and Docker root state. `doctor-cluster` : Diagnose Kubernetes API, nodes, core pods, MetalLB, Traefik, and app health. `inventory-check` : Validate `homelab.yml` with a real YAML parser when available. `secrets-check` : Validate SOPS and age configuration plus committed encrypted secret file structure. `tailnet-policy-check` : Validate the repo-managed Tailscale ACL policy. `ai-check` : Check local AI helper prerequisites, knowledge index, and Ollama availability when the backstage helper is enabled. ### Defensive Security `security-scan` : Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP baseline passive scans for targets in `security/targets.txt`. `security-prepare` : Validate Docker and report prerequisites, then pre-pull scanner images. `security-zap` : Run only OWASP ZAP baseline passive scans against public targets. `security-k8s` : Run kube-bench CIS checks against the Debian kubeadm host. `security-host` : Run a Lynis host audit on Debian and print one highest-risk finding to focus on first. This command is report-only. `security-trivy` : Run only Trivy filesystem and IaC/config scans. `security-secrets` : Run only gitleaks secret scanning against this repo. `security-nuclei` : Run low-rate nuclei HTTP exposure scans against public targets. `security-web` : Run quick HTTPS, status, and defensive-header checks against owned web targets. `security-logs` : Summarize recent Gitea and Traefik logs for common scanner paths, suspicious user agents, auth failures, and HTTP errors. `security-runtime` : Check the Tetragon runtime-detection DaemonSet and print the command for watching recent events. `security-attack-path [namespace] [workload]` : Run a read-only compromise-path drill for a Kubernetes workload. Defaults to the public website deployment. Kyverno hardening policies : `apps/supply-chain-policy` includes audit-mode checks for privileged pods, privilege escalation, hostPath use, resource requests/limits, and mutable image tags. `apps/security-lab` : GitOps-managed internal namespace for intentionally vulnerable practice targets. Use `kubectl port-forward`; do not expose it publicly. ### Maintenance `state-backup` : Create a local tarball backup of OpenTofu state and generated homelab state files. `fix-debian-docker-root` : Repair or migrate the Debian Docker root according to inventory settings. `move-prometheus-stack-workers` : Move Prometheus stack workloads off the control plane. Requires `LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true` because Prometheus PVC data is deleted. `secrets-init` : Initialize local age and SOPS configuration on the Debian host. `ai-index` : Build the local homelab knowledge index for the backstage AI helper. `ollama-setup` : Install or configure Ollama on the Debian host and pull the configured local model. `website-translation-model` : Prepare the website translation model assets. `website-ollama-listen` : Configure the Ollama service listen address for website use. ## COMMON ENVIRONMENT `HOMELAB_INVENTORY_FILE` : Path to the canonical non-secret inventory. Defaults to `./homelab.yml`. `LAB_STATUS_LOG_SINCE` : Log window for status log checks. Defaults to `30m`. `LAB_STATUS_RESTART_THRESHOLD` : Pod restart count that turns the status restart-pressure check into a warning. Defaults to `3`. `LAB_STATUS_DISK_USE_THRESHOLD` : Disk usage percentage that turns the status disk check into a warning. Defaults to `85`. `HOMELAB_STATE_DIR` : Local runtime state directory. Defaults to `${XDG_DATA_HOME:-$HOME/.local/share}/homelab`. `JEANNIE_LOG_DIR` : Directory for command logs. Defaults to `$HOMELAB_STATE_DIR/logs`. `JEANNIE_VERBOSE=true` : Print full step output as it runs. Without this, noisy commands are compact and full output is written to the log file. `LAB_STATUS_DETAILS=true` : Append detailed host, Docker, Kubernetes, Pimox, RPi, Tailscale, and HTTP tables after the normal status cascade. `LAB_STATUS_SHOW_OK_OUTPUT=true` : Show short command output for successful cascade checks. Failed and warning checks already show short output by default. `LAB_PIHOLE_STATUS_QUERY` : DNS name used by `status` and `doctor-preapply` for the Pi-hole resolver smoke test. Defaults to `cloudflare.com`. `SECURITY_REPORT_DIR` : Output directory for defensive security reports. Defaults to `$HOMELAB_STATE_DIR/security-reports`. `SECURITY_TARGETS_FILE` : Public target allowlist for ZAP and nuclei scans. Defaults to `security/targets.txt`. `SECURITY_ZAP_TARGET` : Optional single URL override for `security-zap`. `SECURITY_NUCLEI_TARGET` : Optional single URL override for `security-nuclei`. `SECURITY_WEB_TARGET` : Optional single URL override for `security-web`. `SECURITY_LOG_SINCE` : Log window for `security-logs`. Defaults to `24h`. `lynis` : `security-host` requires Lynis to be installed on the Debian host. Install it explicitly with `sudo apt install lynis`. `LAB_BACKSTAGE_BRAIN_ENABLED=true` : Allows `security-host` to ask the local Ollama/RAG helper to choose the single highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie falls back to the first Lynis warning, then the first suggestion. `LAB_AUTO_APPROVE=false` : Disable automatic OpenTofu approval for apply paths that support confirmation. `LAB_PIMOX_WORKER_COUNT` : Desired Pimox worker count when not supplied by generated cluster topology state. `LAB_SKIP_PREFLIGHT=true` : Skip preflight checks. Use only for targeted troubleshooting. `LAB_PREFLIGHT_ROOT_MIN_FREE_GIB` : Minimum Debian `/` free space for `doctor-preapply`. Default is `10`. `LAB_PREFLIGHT_DATA_MIN_FREE_GIB` : Minimum Debian `/data` free space for `doctor-preapply`. Default is `20`. `LAB_PREFLIGHT_EDGE_MIN_FREE_GIB` : Minimum OCI edge `/` free space for `doctor-preapply`. Default is `2`. `LAB_GITEA_INSTALL_DOCKER=true` : Allow `deploy-gitea` to install Docker through `get.docker.com` when Docker is missing. Default is to fail and ask for host bootstrap. `LAB_RPI_INSTALL_DOCKER=true` : Allow `rpi-services` to install Docker through `get.docker.com` on the Raspberry Pi when Docker is missing. Default is to fail. `TF_VAR_edge_install_docker=true` : Allow the edge OpenTofu stack to install Docker on the OCI host when Docker is missing. Default is to fail. `TF_VAR_haproxy_stats_password` : Required HAProxy stats password for the edge stack. It must be set explicitly and must not use the old default. `LAB_CONFIRM_NUKE=homelab` : Required confirmation for direct `./jeannie nuke`. `LAB_NUKE_DESTROY_PIMOX_WORKERS=true` : Also destroy Pimox worker VMs during `nuke`. `LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true` : Required confirmation for `move-prometheus-stack-workers`. ## FILES `homelab.yml` : Canonical non-secret inventory for hosts, IPs, domains, ports, and the main script name. `.lab/cluster-workers.auto.tfvars.json` : Generated worker topology passed to the cluster stack. `.lab/pimox-workers.tsv` : Generated Pimox worker inventory. `.lab/manual-workers.tsv` : User-managed manual worker inventory. `$HOMELAB_STATE_DIR/logs` : Jeannie command logs. `$HOMELAB_STATE_DIR/tofu-state-backups` : Local OpenTofu and generated state backups. `$HOMELAB_STATE_DIR/security-reports` : Defensive scan reports. `security/targets.txt` : Public target allowlist for defensive web scans. ## EXAMPLES Start a three-worker homelab: ```sh LAB_PIMOX_WORKER_COUNT=3 ./jeannie up ``` Preview all OpenTofu changes: ```sh ./jeannie plan ``` Run focused edge diagnostics: ```sh ./jeannie doctor-edge ``` Run the default defensive scan set: ```sh ./jeannie security-scan ``` Prepare scanner images before the first run: ```sh ./jeannie security-prepare ``` Run a ZAP baseline scan against one URL: ```sh SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap ``` Run a report-only Lynis host audit: ```sh ./jeannie security-host ``` Run the blocking pre-apply doctor directly: ```sh ./jeannie doctor-preapply ``` Create a local state backup: ```sh ./jeannie state-backup ``` Destroy the cluster path intentionally: ```sh LAB_CONFIRM_NUKE=homelab ./jeannie nuke ``` ## EXIT STATUS `0` : Command completed successfully. Non-zero : A check, deployment step, or external tool failed. Check the printed error and the referenced Jeannie log file for full command output. ## SEE ALSO `README.md`, `docs/service-catalog.md`, `docs/runbooks/edge-failures.md`, `docs/runbooks/gitea-failures.md`, `docs/runbooks/cluster-stop-start-failures.md`, `docs/secrets.md`.