470 lines
13 KiB
Markdown
470 lines
13 KiB
Markdown
# JEANNIE(1)
|
|
|
|
## NAME
|
|
|
|
jeannie - operate the personal homelab control pipeline
|
|
|
|
## SYNOPSIS
|
|
|
|
```sh
|
|
./jeannie COMMAND [ARG...]
|
|
```
|
|
|
|
## DESCRIPTION
|
|
|
|
`jeannie` is the main homelab operator script. It deploys and checks the Debian
|
|
control host, Gitea, Raspberry Pi services, Pimox workers, Kubernetes platform
|
|
components, applications, public edge routing, secrets tooling, and local AI
|
|
helpers.
|
|
|
|
Run mutating infrastructure commands from the Debian homelab host. Commands that
|
|
can destroy or rewrite state have explicit confirmation environment variables.
|
|
The canonical script name is `metadata.main_script` in `homelab.yml`.
|
|
|
|
## COMMANDS
|
|
|
|
### Core Lifecycle
|
|
|
|
`up`
|
|
: Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi
|
|
services, runs full preflight and pre-apply doctor checks, then applies Pimox,
|
|
OpenWrt, cluster, platform, apps, and edge stacks.
|
|
|
|
`plan [all|provisioning|cluster|platform|apps|edge]`
|
|
: Run OpenTofu plans without applying changes. The default target is `all`.
|
|
|
|
`rebuild-cluster`
|
|
: Back up current local OpenTofu state, destroy cluster state through the guarded
|
|
nuke path, rebuild Pimox workers, recreate the cluster, and redeploy platform,
|
|
apps, and edge.
|
|
|
|
`stop-cluster`
|
|
: Stop the Kubernetes control-plane runtime and worker VMs without destroying
|
|
cluster state.
|
|
|
|
`start-cluster`
|
|
: Start the Kubernetes control-plane runtime and desired worker VMs.
|
|
|
|
`status`
|
|
: Print a cascade health report for the configured homelab, from bootstrap
|
|
readiness through public website health. Set `LAB_STATUS_DETAILS=true` to append
|
|
the older detailed tables. The cascade includes "what broke" signals for recent
|
|
deployment readiness, pod restarts, node pressure, disk pressure, Traefik
|
|
5xx/404 log evidence, and recent Gitea errors.
|
|
|
|
`capacity`
|
|
: Print a read-only placement report covering Debian memory/disk/Docker usage,
|
|
Kubernetes node and PVC usage, Pimox VM/storage allocation, RPi Docker/disk
|
|
state, and practical placement guidance.
|
|
|
|
`recover-plan`
|
|
: Print the ordered disaster recovery checklist and lightweight prerequisite
|
|
checks for Debian, Gitea, RPi DNS, Pimox, Kubernetes, GitOps apps, and edge
|
|
verification.
|
|
|
|
Prometheus alerts
|
|
: High-signal alert rules are managed as the `apps/homelab-alerts` GitOps app.
|
|
They cover node readiness, restart storms, unavailable deployments, storage
|
|
pressure, Traefik 5xx, and core service visibility.
|
|
|
|
NetworkPolicies
|
|
: Owned application namespaces include egress policies. The website can reach
|
|
DNS, Redis, and the Debian Ollama endpoint; the security lab can reach DNS and
|
|
external web targets while remaining isolated from cluster and LAN ranges.
|
|
|
|
`validate`
|
|
: Run non-mutating repository checks used by the GitOps promotion flow:
|
|
inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu
|
|
formatting, tailnet policy validation, and optional static security checks.
|
|
|
|
`access-audit`
|
|
: Run a read-only access inventory across SSH targets, Git/Gitea remotes,
|
|
Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the
|
|
Gitea runner.
|
|
|
|
`kubeconfig-readonly`
|
|
: Generate a separate local read-only kubeconfig from the repo-managed
|
|
`homelab-access/homelab-readonly` ServiceAccount.
|
|
|
|
`nuke`
|
|
: Destroy Kubernetes state and Pimox worker VMs. Requires
|
|
`LAB_CONFIRM_NUKE=homelab`.
|
|
|
|
### Deploy Targets
|
|
|
|
`apps`
|
|
: Deploy platform applications and build or publish application images when
|
|
needed.
|
|
|
|
`deploy-gitea`
|
|
: Deploy the external Gitea Docker Compose service on the Debian host.
|
|
|
|
`rpi-services`
|
|
: Deploy Pi-hole, Unbound, and Uptime Kuma on the Raspberry Pi.
|
|
|
|
`bootstrap-gitea-repo`
|
|
: Ensure the configured Gitea repository, branch, remotes, and host SSH key are
|
|
ready.
|
|
|
|
`openwrt`
|
|
: Apply the OpenWrt VM stack.
|
|
|
|
### Gitea Operations
|
|
|
|
`backup-gitea`
|
|
: Run a Gitea backup on the Debian host.
|
|
|
|
`drill-restore`
|
|
: Run a read-only restore checklist for Gitea backups, Pi-hole repo-managed
|
|
config, OpenTofu state backup archives, and GitOps rebuild inputs.
|
|
|
|
`drill-gitea-restore`
|
|
: Run the Gitea restore drill workflow.
|
|
|
|
`drill-pihole-restore`
|
|
: Validate that repo-managed Pi-hole restore inputs exist and are non-empty.
|
|
|
|
`install-gitea-runner [TOKEN]`
|
|
: Install or repair the Gitea Actions runner. If `TOKEN` is omitted, the command
|
|
uses the configured token source.
|
|
|
|
### Checks And Doctors
|
|
|
|
`preflight`
|
|
: Run homelab preflight checks. This verifies inventory-derived host settings,
|
|
Docker roots, Tailscale IPs, Gitea reachability, Pimox storage, and OCI edge SSH.
|
|
|
|
`doctor-preapply`
|
|
: Run blocking readiness checks used by `up` before high-blast-radius applies.
|
|
It checks Debian free disk, Docker and containerd, Raspberry Pi Docker
|
|
writability, Pimox storage, OCI edge disk, and Pi-hole DNS.
|
|
|
|
`doctor-versions`
|
|
: Check Kubernetes component version alignment.
|
|
|
|
`doctor-edge`
|
|
: Diagnose public edge routing, Traefik reachability, and Gitea edge exposure.
|
|
|
|
`doctor-gitea`
|
|
: Diagnose Debian Gitea container, local access, edge access, and repository
|
|
health.
|
|
|
|
`doctor-rpi`
|
|
: Diagnose Raspberry Pi services, Pi-hole DNS, Unbound DNS, fallback DNS,
|
|
Uptime Kuma, and Docker root state.
|
|
|
|
`doctor-cluster`
|
|
: Diagnose Kubernetes API, nodes, core pods, MetalLB, Traefik, and app health.
|
|
|
|
`inventory-check`
|
|
: Validate `homelab.yml` with a real YAML parser when available.
|
|
|
|
`secrets-check`
|
|
: Validate SOPS and age configuration plus committed encrypted secret file
|
|
structure.
|
|
|
|
`tailnet-policy-check`
|
|
: Validate the repo-managed Tailscale ACL policy.
|
|
|
|
`ai-check`
|
|
: Check local AI helper prerequisites, knowledge index, and Ollama availability
|
|
when the backstage helper is enabled.
|
|
|
|
### Defensive Security
|
|
|
|
`security-scan`
|
|
: Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP
|
|
baseline passive scans for targets in `security/targets.txt`.
|
|
|
|
`security-prepare`
|
|
: Validate Docker and report prerequisites, then pre-pull scanner images.
|
|
|
|
`security-zap`
|
|
: Run only OWASP ZAP baseline passive scans against public targets.
|
|
|
|
`security-k8s`
|
|
: Run kube-bench CIS checks against the Debian kubeadm host.
|
|
|
|
`security-host`
|
|
: Run a Lynis host audit on Debian and print one highest-risk finding to focus
|
|
on first. This command is report-only.
|
|
|
|
`security-trivy`
|
|
: Run only Trivy filesystem and IaC/config scans.
|
|
|
|
`security-secrets`
|
|
: Run only gitleaks secret scanning against this repo.
|
|
|
|
`security-nuclei`
|
|
: Run low-rate nuclei HTTP exposure scans against public targets.
|
|
|
|
`security-web`
|
|
: Run quick HTTPS, status, and defensive-header checks against owned web
|
|
targets.
|
|
|
|
`security-logs`
|
|
: Summarize recent Gitea and Traefik logs for common scanner paths, suspicious
|
|
user agents, auth failures, and HTTP errors.
|
|
|
|
`security-runtime`
|
|
: Check the Tetragon runtime-detection DaemonSet and print the command for
|
|
watching recent events.
|
|
|
|
`security-attack-path [namespace] [workload]`
|
|
: Run a read-only compromise-path drill for a Kubernetes workload. Defaults to
|
|
the public website deployment.
|
|
|
|
Kyverno hardening policies
|
|
: `apps/supply-chain-policy` includes audit-mode checks for privileged pods,
|
|
privilege escalation, hostPath use, resource requests/limits, and mutable image
|
|
tags.
|
|
|
|
`apps/security-lab`
|
|
: GitOps-managed internal namespace for intentionally vulnerable practice
|
|
targets. Use `kubectl port-forward`; do not expose it publicly.
|
|
|
|
### Maintenance
|
|
|
|
`state-backup`
|
|
: Create a local tarball backup of OpenTofu state and generated homelab state
|
|
files.
|
|
|
|
`fix-debian-docker-root`
|
|
: Repair or migrate the Debian Docker root according to inventory settings.
|
|
|
|
`move-prometheus-stack-workers`
|
|
: Move Prometheus stack workloads off the control plane. Requires
|
|
`LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true` because Prometheus PVC data is deleted.
|
|
|
|
`secrets-init`
|
|
: Initialize local age and SOPS configuration on the Debian host.
|
|
|
|
`ai-index`
|
|
: Build the local homelab knowledge index for the backstage AI helper.
|
|
|
|
`ollama-setup`
|
|
: Install or configure Ollama on the Debian host and pull the configured local
|
|
model.
|
|
|
|
`website-translation-model`
|
|
: Prepare the website translation model assets.
|
|
|
|
`website-ollama-listen`
|
|
: Configure the Ollama service listen address for website use.
|
|
|
|
## COMMON ENVIRONMENT
|
|
|
|
`HOMELAB_INVENTORY_FILE`
|
|
: Path to the canonical non-secret inventory. Defaults to `./homelab.yml`.
|
|
|
|
`LAB_STATUS_LOG_SINCE`
|
|
: Log window for status log checks. Defaults to `30m`.
|
|
|
|
`LAB_STATUS_RESTART_THRESHOLD`
|
|
: Pod restart count that turns the status restart-pressure check into a warning.
|
|
Defaults to `3`.
|
|
|
|
`LAB_STATUS_DISK_USE_THRESHOLD`
|
|
: Disk usage percentage that turns the status disk check into a warning.
|
|
Defaults to `85`.
|
|
|
|
`HOMELAB_STATE_DIR`
|
|
: Local runtime state directory. Defaults to
|
|
`${XDG_DATA_HOME:-$HOME/.local/share}/homelab`.
|
|
|
|
`JEANNIE_LOG_DIR`
|
|
: Directory for command logs. Defaults to `$HOMELAB_STATE_DIR/logs`.
|
|
|
|
`JEANNIE_VERBOSE=true`
|
|
: Print full step output as it runs. Without this, noisy commands are compact and
|
|
full output is written to the log file.
|
|
|
|
`LAB_STATUS_DETAILS=true`
|
|
: Append detailed host, Docker, Kubernetes, Pimox, RPi, Tailscale, and HTTP
|
|
tables after the normal status cascade.
|
|
|
|
`LAB_STATUS_SHOW_OK_OUTPUT=true`
|
|
: Show short command output for successful cascade checks. Failed and warning
|
|
checks already show short output by default.
|
|
|
|
`LAB_PIHOLE_STATUS_QUERY`
|
|
: DNS name used by `status` and `doctor-preapply` for the Pi-hole resolver
|
|
smoke test. Defaults to `cloudflare.com`.
|
|
|
|
`SECURITY_REPORT_DIR`
|
|
: Output directory for defensive security reports. Defaults to
|
|
`$HOMELAB_STATE_DIR/security-reports`.
|
|
|
|
`SECURITY_TARGETS_FILE`
|
|
: Public target allowlist for ZAP and nuclei scans. Defaults to
|
|
`security/targets.txt`.
|
|
|
|
`SECURITY_ZAP_TARGET`
|
|
: Optional single URL override for `security-zap`.
|
|
|
|
`SECURITY_NUCLEI_TARGET`
|
|
: Optional single URL override for `security-nuclei`.
|
|
|
|
`SECURITY_WEB_TARGET`
|
|
: Optional single URL override for `security-web`.
|
|
|
|
`SECURITY_LOG_SINCE`
|
|
: Log window for `security-logs`. Defaults to `24h`.
|
|
|
|
`lynis`
|
|
: `security-host` requires Lynis to be installed on the Debian host. Install it
|
|
explicitly with `sudo apt install lynis`.
|
|
|
|
`LAB_BACKSTAGE_BRAIN_ENABLED=true`
|
|
: Allows `security-host` to ask the local Ollama/RAG helper to choose the single
|
|
highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie
|
|
falls back to the first Lynis warning, then the first suggestion.
|
|
|
|
`LAB_AUTO_APPROVE=false`
|
|
: Disable automatic OpenTofu approval for apply paths that support confirmation.
|
|
|
|
`LAB_PIMOX_WORKER_COUNT`
|
|
: Desired Pimox worker count when not supplied by generated cluster topology
|
|
state.
|
|
|
|
`LAB_SKIP_PREFLIGHT=true`
|
|
: Skip preflight checks. Use only for targeted troubleshooting.
|
|
|
|
`LAB_PREFLIGHT_ROOT_MIN_FREE_GIB`
|
|
: Minimum Debian `/` free space for `doctor-preapply`. Default is `10`.
|
|
|
|
`LAB_PREFLIGHT_DATA_MIN_FREE_GIB`
|
|
: Minimum Debian `/data` free space for `doctor-preapply`. Default is `20`.
|
|
|
|
`LAB_PREFLIGHT_EDGE_MIN_FREE_GIB`
|
|
: Minimum OCI edge `/` free space for `doctor-preapply`. Default is `2`.
|
|
|
|
`LAB_GITEA_INSTALL_DOCKER=true`
|
|
: Allow `deploy-gitea` to install Docker through `get.docker.com` when Docker is
|
|
missing. Default is to fail and ask for host bootstrap.
|
|
|
|
`LAB_RPI_INSTALL_DOCKER=true`
|
|
: Allow `rpi-services` to install Docker through `get.docker.com` on the
|
|
Raspberry Pi when Docker is missing. Default is to fail.
|
|
|
|
`TF_VAR_edge_install_docker=true`
|
|
: Allow the edge OpenTofu stack to install Docker on the OCI host when Docker is
|
|
missing. Default is to fail.
|
|
|
|
`TF_VAR_haproxy_stats_password`
|
|
: Required HAProxy stats password for the edge stack. It must be set explicitly
|
|
and must not use the old default.
|
|
|
|
`LAB_CONFIRM_NUKE=homelab`
|
|
: Required confirmation for direct `./jeannie nuke`.
|
|
|
|
`LAB_NUKE_DESTROY_PIMOX_WORKERS=true`
|
|
: Also destroy Pimox worker VMs during `nuke`.
|
|
|
|
`LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true`
|
|
: Required confirmation for `move-prometheus-stack-workers`.
|
|
|
|
## FILES
|
|
|
|
`homelab.yml`
|
|
: Canonical non-secret inventory for hosts, IPs, domains, ports, and the main
|
|
script name.
|
|
|
|
`.lab/cluster-workers.auto.tfvars.json`
|
|
: Generated worker topology passed to the cluster stack.
|
|
|
|
`.lab/pimox-workers.tsv`
|
|
: Generated Pimox worker inventory.
|
|
|
|
`.lab/manual-workers.tsv`
|
|
: User-managed manual worker inventory.
|
|
|
|
`$HOMELAB_STATE_DIR/logs`
|
|
: Jeannie command logs.
|
|
|
|
`$HOMELAB_STATE_DIR/tofu-state-backups`
|
|
: Local OpenTofu and generated state backups.
|
|
|
|
`$HOMELAB_STATE_DIR/security-reports`
|
|
: Defensive scan reports.
|
|
|
|
`security/targets.txt`
|
|
: Public target allowlist for defensive web scans.
|
|
|
|
## EXAMPLES
|
|
|
|
Start a three-worker homelab:
|
|
|
|
```sh
|
|
LAB_PIMOX_WORKER_COUNT=3 ./jeannie up
|
|
```
|
|
|
|
Preview all OpenTofu changes:
|
|
|
|
```sh
|
|
./jeannie plan
|
|
```
|
|
|
|
Run focused edge diagnostics:
|
|
|
|
```sh
|
|
./jeannie doctor-edge
|
|
```
|
|
|
|
Run the default defensive scan set:
|
|
|
|
```sh
|
|
./jeannie security-scan
|
|
```
|
|
|
|
Prepare scanner images before the first run:
|
|
|
|
```sh
|
|
./jeannie security-prepare
|
|
```
|
|
|
|
Run a ZAP baseline scan against one URL:
|
|
|
|
```sh
|
|
SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap
|
|
```
|
|
|
|
Run a report-only Lynis host audit:
|
|
|
|
```sh
|
|
./jeannie security-host
|
|
```
|
|
|
|
Run the blocking pre-apply doctor directly:
|
|
|
|
```sh
|
|
./jeannie doctor-preapply
|
|
```
|
|
|
|
Create a local state backup:
|
|
|
|
```sh
|
|
./jeannie state-backup
|
|
```
|
|
|
|
Destroy the cluster path intentionally:
|
|
|
|
```sh
|
|
LAB_CONFIRM_NUKE=homelab ./jeannie nuke
|
|
```
|
|
|
|
## EXIT STATUS
|
|
|
|
`0`
|
|
: Command completed successfully.
|
|
|
|
Non-zero
|
|
: A check, deployment step, or external tool failed. Check the printed error and
|
|
the referenced Jeannie log file for full command output.
|
|
|
|
## SEE ALSO
|
|
|
|
`README.md`, `docs/service-catalog.md`, `docs/runbooks/edge-failures.md`,
|
|
`docs/runbooks/gitea-failures.md`,
|
|
`docs/runbooks/cluster-stop-start-failures.md`, `docs/secrets.md`.
|