# JEANNIE(1)

## NAME

jeannie - operate the personal homelab control pipeline

## SYNOPSIS

```sh
./jeannie COMMAND [ARG...]
```

## DESCRIPTION

`jeannie` is the main homelab operator script. It deploys and checks the Debian
control host, Gitea, Raspberry Pi services, Pimox workers, Kubernetes platform
components, applications, public edge routing, secrets tooling, and local AI
helpers.

Run mutating infrastructure commands from the Debian homelab host. Commands that
can destroy or rewrite state have explicit confirmation environment variables.
The canonical script name is `metadata.main_script` in `homelab.yml`.

## COMMANDS

Run `./jeannie help` for the short in-terminal command reference. It is grouped
in the same order as the pipeline: inventory and planning, bootstrap, cluster
lifecycle, reports, observability, recovery, doctors, access, apps, security,
AI, and destructive commands.

### Core Lifecycle

`help`
: Print the grouped command reference. The same output is shown for `-h`,
`--help`, or no command.

`up`
: Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi
services, runs full preflight and pre-apply doctor checks, then applies Pimox,
OpenWrt, cluster, platform, apps, and edge stacks.

`plan [all|provisioning|cluster|platform|apps|edge]`
: Run OpenTofu plans without applying changes. The default target is `all`.

`rebuild-cluster`
: Back up current local OpenTofu state, destroy cluster state through the guarded
nuke path, rebuild Pimox workers, recreate the cluster, and redeploy platform,
apps, and edge.

`stop-cluster`
: Stop the Kubernetes control-plane runtime and worker VMs without destroying
cluster state.

`start-cluster`
: Start the Kubernetes control-plane runtime and desired worker VMs.

`status`
: Print a cascade health report for the configured homelab, from bootstrap
readiness through public website health. Set `LAB_STATUS_DETAILS=true` to append
the older detailed tables. The cascade includes "what broke" signals for recent
deployment readiness, pod restarts, node pressure, disk pressure, Traefik
5xx/404 log evidence, and recent Gitea errors.

`capacity`
: Print a compact read-only placement report covering Debian, Kubernetes,
Pimox, RPi, and placement signals. Use `capacity --details`, `--json`, `--only
problems`, or `--verbose` for the older full dump.

`capacity-advisor`
: Interpret capacity, resource-budget, and control-plane placement signals to
recommend whether to add Pimox VMs, taint Debian, or fix resource definitions
first.

`capacity-limits [namespace]`
: Recommend Kubernetes `resources.requests` and `resources.limits` YAML snippets
for containers missing them. Uses `infra/resource-recommendations.yml` and does
not edit manifests.

`recover-plan`
: Print the ordered disaster recovery checklist and lightweight prerequisite
checks for Debian, Gitea, RPi DNS, Pimox, Kubernetes, GitOps apps, and edge
verification.

`recover-power [--dry-run]`
: Run the post-outage recovery sequence in dependency order: Debian runtime,
Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and edge/public checks.

`scorecard`
: Roll up backup readiness, GitOps health, DNS/RPi health, cluster health, edge
health, capacity pressure, security posture, public certificates, inventory, and
docs freshness into a compact pass/warn/fail report.

Report renderer
: Report-oriented commands can share `scripts/report-render`, which reads status
TSV rows and prints compact grouped summaries, detailed output, or JSON. Rows
may include separate `fix`, `graph`, and `query` fields for Grafana-backed
checks.

`gitops-status`
: Print a focused Argo CD status view with application sync/health, out-of-sync
or degraded apps, repository secret presence, recent events, and recent
repo-server/application-controller errors.

`grafana-dashboards {list|apply}`
: List or rebuild repo-provisioned Grafana dashboard ConfigMaps from
`bootstrap/platform/grafana-dashboards`.

`cert-check`
: Verify public DNS, TLS certificate expiry, public website/Gitea HTTP status,
and DuckDNS-to-OCI edge IP drift from the canonical inventory.

`release-snapshot`
: Write a timestamped pre-change report with Git state, Kubernetes/Argo CD/Helm
state, workload images, public URL status, and latest Gitea/OpenTofu backup
pointers.

`impact [--since REF] [PATH...]`
: Explain which homelab areas are affected by changed files or explicit paths,
including risk, touched services, validation commands, and apply path.

`review-last-change [REF]`
: Summarize a commit, list touched files, run impact analysis, and print generic
validation plus rollback hints. Defaults to `HEAD`.

`backup-status`
: Report freshness for Gitea backups, OpenTofu state backups, restore drill
reports, and repo-managed Pi-hole restore inputs. The main `status` cascade
includes this as a warning signal.

`synthetic-checks`
: Run end-to-end probes for Gitea HTTP/SSH, registry API, Pi-hole DNS, Traefik
LAN HTTP, and temporary Kubernetes DNS/ingress pods. Registry push/pull is
opt-in through `LAB_SYNTHETIC_REGISTRY_PUSH=true`.

`resource-budget`
: Check the report-only resource policy in `infra/resource-budgets.yml` against
Kubernetes pod requests/limits and Debian disk/memory signals. Use
`resource-budget --details` for the full pod/container list.

`control-plane {status|taint|untaint|pods}`
: Show or manage the Debian Kubernetes control-plane scheduling taint. Use
`control-plane taint` after Pimox workers are stable to keep normal app/data
plane pods off the Debian host.

`explain <read-only-command> [args...]`
: Explain warnings, failures, and corrective findings from Jeannie output,
including the meaning, diagnostic commands, whether a Jeannie fix command
already exists, risk, and auto-fix safety. This command is read-only and refuses
mutating commands. Use `explain scorecard [--all]` for scorecard-specific
category explanations, or pipe captured output into `explain output`.

`artifact-cache {status|up|down|instructions}`
: Manage the optional Debian-host artifact cache stack for apt packages and
Docker Hub pull-through image caching.

`blockchain-devnet {status|up|down|logs|rpc}`
: Manage the optional local Ethereum Anvil devnet stack under
`infra/blockchain-devnet`. This is for local learning only; never use real seed
phrases or mainnet keys.

`blockchain-test [forge args...]`
: Run the Foundry smart contract tests in `labs/blockchain`. Extra arguments are
passed through to `forge test`.

`blockchain-wallet {instructions|new|address|sign-message MESSAGE}`
: Practice dev wallet generation, address derivation, and message signing with
Foundry Cast. Use disposable Anvil/dev keys only.

`golden-ledger {show|check}`
: Show or validate the Pimox golden image version ledger.

`route-inventory`
: Report Kubernetes ingress hosts, paths, backend services, TLS coverage, and
visible Uptime Kuma monitor coverage.

`workers <command>`
: Manage Kubernetes/Pimox worker lifecycle operations. Commands include `list`,
`start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and `rebalance`.

`change-journal {list|path}`
: List local journal entries written before risky commands, or print the journal
directory path.

`map`
: Print a dependency map from the canonical inventory. Use `map --dot` for a
Graphviz DOT graph.

Prometheus alerts
: High-signal alert rules are managed as the `apps/homelab-alerts` GitOps app.
They cover node readiness, restart storms, unavailable deployments, storage
pressure, Traefik 5xx, and core service visibility.

NetworkPolicies
: Owned application namespaces include egress policies. The website can reach
DNS, Redis, and the Debian Ollama endpoint; the security lab can reach DNS and
external web targets while remaining isolated from cluster and LAN ranges.

`validate`
: Run non-mutating repository checks used by the GitOps promotion flow:
inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu
formatting, tailnet policy validation, and optional static security checks.

`access-audit`
: Run a read-only access inventory across SSH targets, Git/Gitea remotes,
Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the
Gitea runner.

`kubeconfig-readonly`
: Generate a separate local read-only kubeconfig from the repo-managed
`homelab-access/homelab-readonly` ServiceAccount.

`nuke`
: Destroy Kubernetes state and Pimox worker VMs. Requires
`LAB_CONFIRM_NUKE=homelab`.

### Deploy Targets

`apps`
: Deploy platform applications and build or publish application images when
needed.

`deploy-gitea`
: Deploy the external Gitea Docker Compose service on the Debian host.

`rpi-services`
: Deploy Pi-hole, Unbound, and Uptime Kuma on the Raspberry Pi.

`bootstrap-gitea-repo`
: Ensure the configured Gitea repository, branch, remotes, and host SSH key are
ready.

`openwrt`
: Apply the OpenWrt VM stack.

### Gitea Operations

`backup-gitea`
: Run a Gitea backup on the Debian host.

`drill-restore`
: Run a read-only restore checklist for Gitea backups, Pi-hole repo-managed
config, OpenTofu state backup archives, and GitOps rebuild inputs.

`drill-gitea-restore`
: Run the Gitea restore drill workflow.

`drill-pihole-restore`
: Validate that repo-managed Pi-hole restore inputs exist and are non-empty.

`install-gitea-runner [TOKEN]`
: Install or repair the Gitea Actions runner. If `TOKEN` is omitted, the command
uses the configured token source.

### Checks And Doctors

`preflight`
: Run homelab preflight checks. This verifies inventory-derived host settings,
Docker roots, Tailscale IPs, Gitea reachability, Pimox storage, and OCI edge SSH.

`doctor-preapply`
: Run blocking readiness checks used by `up` before high-blast-radius applies.
It checks Debian free disk, Docker and containerd, Raspberry Pi Docker
writability, Pimox storage, OCI edge disk, and Pi-hole DNS.

`doctor-versions`
: Check Kubernetes component version alignment.

`doctor-edge`
: Diagnose public edge routing, Traefik reachability, and Gitea edge exposure.

`doctor-gitea`
: Diagnose Debian Gitea container, local access, edge access, and repository
health.

`doctor-rpi`
: Diagnose Raspberry Pi services, Pi-hole DNS, Unbound DNS, fallback DNS,
Uptime Kuma, and Docker root state.

`doctor-cluster`
: Diagnose Kubernetes API, nodes, core pods, MetalLB, Traefik, and app health.

`inventory-check`
: Validate `homelab.yml` with a real YAML parser when available.

`secrets-check`
: Validate SOPS and age configuration plus committed encrypted secret file
structure.

`tailnet-policy-check`
: Validate the repo-managed Tailscale ACL policy.

`ai-check`
: Check local AI helper prerequisites, knowledge index, and Ollama availability
when the backstage helper is enabled.

`ask QUESTION...`
: Search the local homelab knowledge index for relevant docs, runbooks, scripts,
and commands. Set `LAB_AI_ASK_LLM=true` to ask Ollama after retrieval.

### Defensive Security

`security-scan`
: Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP
baseline passive scans for targets in `security/targets.txt`.

`security-prepare`
: Validate Docker and report prerequisites, then pre-pull scanner images.

`security-zap`
: Run only OWASP ZAP baseline passive scans against public targets.

`security-k8s`
: Run kube-bench CIS checks against the Debian kubeadm host.

`security-host`
: Run a Lynis host audit on Debian and print one highest-risk finding to focus
on first. This command is report-only.

`security-trivy`
: Run only Trivy filesystem and IaC/config scans.

`security-secrets`
: Run only gitleaks secret scanning against this repo.

`security-nuclei`
: Run low-rate nuclei HTTP exposure scans against public targets.

`security-web`
: Run quick HTTPS, status, and defensive-header checks against owned web
targets.

`security-logs`
: Summarize recent Gitea and Traefik logs for common scanner paths, suspicious
user agents, auth failures, and HTTP errors.

`security-runtime`
: Check the Tetragon runtime-detection DaemonSet and print the command for
watching recent events.

`security-attack-path [namespace] [workload]`
: Run a read-only compromise-path drill for a Kubernetes workload. Defaults to
the public website deployment.

Kyverno hardening policies
: `apps/supply-chain-policy` includes audit-mode checks for privileged pods,
privilege escalation, hostPath use, resource requests/limits, and mutable image
tags.

`apps/security-lab`
: GitOps-managed internal namespace for intentionally vulnerable practice
targets. Use `kubectl port-forward`; do not expose it publicly.

### Maintenance

`state-backup`
: Create a local tarball backup of OpenTofu state and generated homelab state
files.

`fix-debian-docker-root`
: Repair or migrate the Debian Docker root according to inventory settings.

`move-prometheus-stack-workers`
: Move Prometheus stack workloads off the control plane. Requires
`LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true` because Prometheus PVC data is deleted.

`secrets-init`
: Initialize local age and SOPS configuration on the Debian host.

`ai-index`
: Build the local homelab knowledge index for the backstage AI helper.

`ollama-setup`
: Install or configure Ollama on the Debian host and pull the configured local
model.

`website-translation-model`
: Prepare the website translation model assets.

`website-ollama-listen`
: Configure the Ollama service listen address for website use.

## COMMON ENVIRONMENT

`HOMELAB_INVENTORY_FILE`
: Path to the canonical non-secret inventory. Defaults to `./homelab.yml`.

`LAB_STATUS_LOG_SINCE`
: Log window for status log checks. Defaults to `30m`.

`LAB_STATUS_RESTART_THRESHOLD`
: Pod restart count that turns the status restart-pressure check into a warning.
Defaults to `3`.

`LAB_STATUS_DISK_USE_THRESHOLD`
: Disk usage percentage that turns the status disk check into a warning.
Defaults to `85`.

`HOMELAB_STATE_DIR`
: Local runtime state directory. Defaults to
`${XDG_DATA_HOME:-$HOME/.local/share}/homelab`.

`JEANNIE_LOG_DIR`
: Directory for command logs. Defaults to `$HOMELAB_STATE_DIR/logs`.

`JEANNIE_VERBOSE=true`
: Print full step output as it runs. Without this, noisy commands are compact and
full output is written to the log file.

`LAB_STATUS_DETAILS=true`
: Append detailed host, Docker, Kubernetes, Pimox, RPi, Tailscale, and HTTP
tables after the normal status cascade.

`LAB_STATUS_SHOW_OK_OUTPUT=true`
: Show short command output for successful cascade checks. Failed and warning
checks already show short output by default.

`LAB_PIHOLE_STATUS_QUERY`
: DNS name used by `status` and `doctor-preapply` for the Pi-hole resolver
smoke test. Defaults to `cloudflare.com`.

`SECURITY_REPORT_DIR`
: Output directory for defensive security reports. Defaults to
`$HOMELAB_STATE_DIR/security-reports`.

`SECURITY_TARGETS_FILE`
: Public target allowlist for ZAP and nuclei scans. Defaults to
`security/targets.txt`.

`SECURITY_ZAP_TARGET`
: Optional single URL override for `security-zap`.

`SECURITY_NUCLEI_TARGET`
: Optional single URL override for `security-nuclei`.

`SECURITY_WEB_TARGET`
: Optional single URL override for `security-web`.

`SECURITY_LOG_SINCE`
: Log window for `security-logs`. Defaults to `24h`.

`lynis`
: `security-host` requires Lynis to be installed on the Debian host. Install it
explicitly with `sudo apt install lynis`.

`LAB_BACKSTAGE_BRAIN_ENABLED=true`
: Allows `security-host` to ask the local Ollama/RAG helper to choose the single
highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie
falls back to the first Lynis warning, then the first suggestion.

`LAB_AUTO_APPROVE=false`
: Disable automatic OpenTofu approval for apply paths that support confirmation.

`LAB_PIMOX_WORKER_COUNT`
: Desired Pimox worker count when not supplied by generated cluster topology
state.

`LAB_SKIP_PREFLIGHT=true`
: Skip preflight checks. Use only for targeted troubleshooting.

`LAB_PREFLIGHT_ROOT_MIN_FREE_GIB`
: Minimum Debian `/` free space for `doctor-preapply`. Default is `10`.

`LAB_PREFLIGHT_DATA_MIN_FREE_GIB`
: Minimum Debian `/data` free space for `doctor-preapply`. Default is `20`.

`LAB_PREFLIGHT_EDGE_MIN_FREE_GIB`
: Minimum OCI edge `/` free space for `doctor-preapply`. Default is `2`.

`LAB_GITEA_INSTALL_DOCKER=true`
: Allow `deploy-gitea` to install Docker through `get.docker.com` when Docker is
missing. Default is to fail and ask for host bootstrap.

`LAB_RPI_INSTALL_DOCKER=true`
: Allow `rpi-services` to install Docker through `get.docker.com` on the
Raspberry Pi when Docker is missing. Default is to fail.

`TF_VAR_edge_install_docker=true`
: Allow the edge OpenTofu stack to install Docker on the OCI host when Docker is
missing. Default is to fail.

`TF_VAR_haproxy_stats_password`
: Required HAProxy stats password for the edge stack. It must be set explicitly
and must not use the old default.

`LAB_CONFIRM_NUKE=homelab`
: Required confirmation for direct `./jeannie nuke`.

`LAB_NUKE_DESTROY_PIMOX_WORKERS=true`
: Also destroy Pimox worker VMs during `nuke`.

`LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true`
: Required confirmation for `move-prometheus-stack-workers`.

## FILES

`homelab.yml`
: Canonical non-secret inventory for hosts, IPs, domains, ports, and the main
script name.

`.lab/cluster-workers.auto.tfvars.json`
: Generated worker topology passed to the cluster stack.

`.lab/pimox-workers.tsv`
: Generated Pimox worker inventory.

`.lab/manual-workers.tsv`
: User-managed manual worker inventory.

`$HOMELAB_STATE_DIR/logs`
: Jeannie command logs.

`$HOMELAB_STATE_DIR/tofu-state-backups`
: Local OpenTofu and generated state backups.

`$HOMELAB_STATE_DIR/security-reports`
: Defensive scan reports.

`security/targets.txt`
: Public target allowlist for defensive web scans.

## EXAMPLES

Start a three-worker homelab:

```sh
LAB_PIMOX_WORKER_COUNT=3 ./jeannie up
```

Preview all OpenTofu changes:

```sh
./jeannie plan
```

Run focused edge diagnostics:

```sh
./jeannie doctor-edge
```

Run the default defensive scan set:

```sh
./jeannie security-scan
```

Prepare scanner images before the first run:

```sh
./jeannie security-prepare
```

Run a ZAP baseline scan against one URL:

```sh
SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap
```

Run a report-only Lynis host audit:

```sh
./jeannie security-host
```

Run the blocking pre-apply doctor directly:

```sh
./jeannie doctor-preapply
```

Create a local state backup:

```sh
./jeannie state-backup
```

Destroy the cluster path intentionally:

```sh
LAB_CONFIRM_NUKE=homelab ./jeannie nuke
```

## EXIT STATUS

`0`
: Command completed successfully.

Non-zero
: A check, deployment step, or external tool failed. Check the printed error and
the referenced Jeannie log file for full command output.

## SEE ALSO

`README.md`, `docs/service-catalog.md`, `docs/runbooks/edge-failures.md`,
`docs/runbooks/gitea-failures.md`,
`docs/runbooks/cluster-stop-start-failures.md`, `docs/secrets.md`.