23 KiB

Raw Blame History

JEANNIE(1)

NAME

jeannie - operate the personal homelab control pipeline

SYNOPSIS

./jeannie COMMAND [ARG...]

DESCRIPTION

jeannie is the main homelab operator script. It deploys and checks the Debian control host, Gitea, Raspberry Pi services, Pimox workers, Kubernetes platform components, applications, public edge routing, secrets tooling, and local AI helpers.

Run mutating infrastructure commands from the Debian homelab host. Commands that can destroy or rewrite state have explicit confirmation environment variables. The canonical script name is metadata.main_script in homelab.yml.

COMMANDS

Run ./jeannie help for the short in-terminal command reference. It is grouped in the same order as the pipeline: inventory and planning, bootstrap, cluster lifecycle, reports, observability, recovery, doctors, access, apps, security, AI, and destructive commands.

Core Lifecycle

help: Print the grouped command reference. The same output is shown for -h, --help, or no command.
up: Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi services, runs full preflight and pre-apply doctor checks, then applies Pimox, OpenWrt, cluster, platform, apps, and edge stacks.
plan [all|provisioning|cluster|platform|apps|edge]: Run OpenTofu plans without applying changes. The default target is all.
rebuild-cluster: Back up current local OpenTofu state, destroy cluster state through the guarded nuke path, rebuild Pimox workers, recreate the cluster, and redeploy platform, apps, and edge.
stop-cluster: Stop the Kubernetes control-plane runtime and worker VMs without destroying cluster state.
start-cluster: Start the Kubernetes control-plane runtime and desired worker VMs.
status: Print a problems-first cascade health report for the configured homelab, from bootstrap readiness through public website health. By default it only shows failures and warnings. Use status --all for every check, status --details for row details, status --json for structured output, or status --verbose for the older detailed tables. The cascade includes "what broke" signals for recent deployment readiness, pod restarts, node pressure, disk pressure, Traefik 5xx/404 log evidence, and recent Gitea errors.
status --heal-plan: Build a guarded self-heal plan from current status findings. This is read-only and may include cited RAG context when the local index is available.
status --heal: Build the same plan and dry-run the single highest-impact auto-eligible action. Use heal apply --yes to actually run it.
heal {plan|apply}: Plan or apply guarded self-healing actions. apply executes at most one highest-impact auto-eligible action allowed by infra/agent-sandbox/policy.tsv; destructive actions remain blocked. Re-run status before the next heal cycle. After an action runs, Jeannie waits for the target condition to clear and reports healed or still failing. Failed actions are recorded in the homelab state directory and placed on a short cooldown; during cooldown the next plan escalates to diagnostics instead of repeating the same restart.
capacity: Print a compact read-only placement report covering Debian, Kubernetes, Pimox, RPi, and placement signals. Use capacity --details, --json, --only problems, or --verbose for the older full dump.
capacity-advisor: Interpret capacity, resource-budget, and control-plane placement signals to recommend whether to add Pimox VMs, taint Debian, or fix resource definitions first.
capacity-limits [namespace]: Recommend Kubernetes resources.requests and resources.limits YAML snippets for containers missing them. Uses infra/resource-recommendations.yml and does not edit manifests.
recover-plan: Print the ordered disaster recovery checklist and lightweight prerequisite checks for Debian, Gitea, RPi DNS, Pimox, Kubernetes, GitOps apps, and edge verification.
recover-power [--dry-run]: Run the post-outage recovery sequence in dependency order: Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and edge/public checks.
incident {list|replay|triage}: Classify incident text, replay known fixtures, and print the likely hypothesis, runbook, next read-only commands, and actions blocked until explicitly approved.
safety-case [--since REF] [PATH...]: Generate a read-only safety case for a Git diff or explicit paths, including risk, blast radius, required validation, rollback, and forbidden actions.
agent-sandbox {check|policy}: Classify proposed commands against repo-managed agent action policy. The first version is read-only and reports allow, approval, or deny.
scorecard: Roll up backup readiness, GitOps health, DNS/RPi health, cluster health, edge health, capacity pressure, security posture, public certificates, inventory, and docs freshness into a compact pass/warn/fail report.
Report renderer: Report-oriented commands share scripts/report-ui and scripts/report-render. The UI module emits status TSV rows, and the renderer prints compact grouped summaries, detailed output, or JSON. Rows may include separate fix, graph, and query fields for Grafana-backed checks. Default doctor/status output is problems-first; pass --all, --details, --json, or --verbose when more context is needed.
gitops-status: Print a focused Argo CD status view with application sync/health, out-of-sync or degraded apps, repository secret presence, recent events, and recent repo-server/application-controller errors.
promote {plan|validate|rollback APP}: Print the canary promotion path, run local required promotion gates, or print rollback steps for an app.
grafana-dashboards {list|apply}: List or rebuild repo-provisioned Grafana dashboard ConfigMaps from bootstrap/platform/grafana-dashboards.
cert-check: Verify public DNS, TLS certificate expiry, public website/Gitea HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory.
release-snapshot: Write a timestamped pre-change report with Git state, Kubernetes/Argo CD/Helm state, workload images, public URL status, and latest Gitea/OpenTofu backup pointers.
impact [--since REF] [PATH...]: Explain which homelab areas are affected by changed files or explicit paths, including risk, touched services, validation commands, and apply path.
review-last-change [REF]: Summarize a commit, list touched files, run impact analysis, and print generic validation plus rollback hints. Defaults to HEAD.
backup-status: Report freshness for Gitea backups, OpenTofu state backups, restore drill reports, and repo-managed Pi-hole restore inputs. The main status cascade includes this as a warning signal.
synthetic-checks: Run end-to-end probes for Gitea HTTP/SSH, registry API, Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. Registry push/pull is opt-in through LAB_SYNTHETIC_REGISTRY_PUSH=true.
resource-budget: Check the report-only resource policy in infra/resource-budgets.yml against Kubernetes pod requests/limits and Debian disk/memory signals. Use resource-budget --details for the full pod/container list.
control-plane {status|taint|untaint|pods}: Show or manage the Debian Kubernetes control-plane scheduling taint. Use control-plane taint after Pimox workers are stable to keep normal app/data plane pods off the Debian host.
explain <read-only-command> [args...]: Explain warnings, failures, and corrective findings from Jeannie output, including the meaning, diagnostic commands, whether a Jeannie fix command already exists, risk, and auto-fix safety. This command is read-only and refuses mutating commands. Use explain scorecard [--all] for scorecard-specific category explanations, use it directly with helpers such as capacity-advisor, capacity-limits, impact, and review-last-change, or pipe captured output into explain output.
artifact-cache {status|up|down|instructions}: Manage the optional Debian-host artifact cache stack for apt packages and Docker Hub pull-through image caching.
blockchain-devnet {status|up|down|logs|rpc}: Manage the optional local Ethereum Anvil devnet stack under infra/blockchain-devnet. This is for local learning only; never use real seed phrases or mainnet keys.
blockchain-test [forge args...]: Run the Foundry smart contract tests in labs/blockchain. Extra arguments are passed through to forge test.
blockchain-wallet {instructions|new|address|sign-message MESSAGE}: Practice dev wallet generation, address derivation, and message signing with Foundry Cast. Use disposable Anvil/dev keys only.
golden-ledger {show|check}: Show or validate the Pimox golden image version ledger.
route-inventory: Report Kubernetes ingress hosts, paths, backend services, TLS coverage, and visible Uptime Kuma monitor coverage.
workers <command>: Manage Kubernetes/Pimox worker lifecycle operations. Commands include list, tailnet, start, stop, restart, drain, uncordon, recreate-plan, and rebalance.
change-journal {list|path}: List local journal entries written before risky commands, or print the journal directory path.
map: Print a dependency map from the canonical inventory. Use map --dot for a Graphviz DOT graph.
Prometheus alerts: High-signal alert rules are managed as the apps/homelab-alerts GitOps app. They cover node readiness, restart storms, unavailable deployments, storage pressure, Traefik 5xx, and core service visibility.
NetworkPolicies: Owned application namespaces include egress policies. The website can reach DNS, Redis, and the Debian Ollama endpoint; the security lab can reach DNS and external web targets while remaining isolated from cluster and LAN ranges.
validate: Run non-mutating repository checks used by the GitOps promotion flow: inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu formatting, tailnet policy validation, and optional static security checks.
access-audit: Run a read-only access inventory across SSH targets, Git/Gitea remotes, Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the Gitea runner.
kubeconfig-readonly: Generate a separate local read-only kubeconfig from the repo-managed homelab-access/homelab-readonly ServiceAccount.
nuke: Destroy Kubernetes state and Pimox worker VMs. Requires LAB_CONFIRM_NUKE=homelab.

Deploy Targets

apps: Deploy platform applications and build or publish application images when needed.
deploy-gitea: Deploy the external Gitea Docker Compose service on the Debian host.
rpi-services: Deploy Pi-hole, Unbound, and Uptime Kuma on the Raspberry Pi.
bootstrap-gitea-repo: Ensure the configured Gitea repository, branch, remotes, and host SSH key are ready.
openwrt: Apply the OpenWrt VM stack.

Gitea Operations

backup-gitea: Run a Gitea backup on the Debian host.
drill-restore: Run a read-only restore checklist for Gitea backups, Pi-hole repo-managed config, OpenTofu state backup archives, and GitOps rebuild inputs.
drill-gitea-restore: Run the Gitea restore drill workflow.
drill-pihole-restore: Validate that repo-managed Pi-hole restore inputs exist and are non-empty.
install-gitea-runner [TOKEN]: Install or repair the Gitea Actions runner. If TOKEN is omitted, the command uses the configured token source.

Checks And Doctors

preflight: Run homelab preflight checks. This verifies inventory-derived host settings, Docker roots, Tailscale IPs, Gitea reachability, Pimox storage, and OCI edge SSH.
doctor-preapply: Run blocking readiness checks used by up before high-blast-radius applies. It checks Debian free disk, Docker and containerd, Raspberry Pi Docker writability, Pimox storage, OCI edge disk, and Pi-hole DNS.
doctor-versions [--all|--details|--verbose|--json]: Check Kubernetes component version alignment. Default output shows only warnings and failures; --verbose prints the old version table.
doctor-edge [--all|--details|--verbose|--json]: Diagnose public edge routing, Traefik reachability, and Gitea edge exposure. Default output shows only warnings and failures.
doctor-gitea [--all|--details|--verbose|--json]: Diagnose Debian Gitea container, local access, edge access, and repository health. Default output shows only warnings and failures.
doctor-rpi [--all|--details|--verbose|--json]: Diagnose Raspberry Pi services, Pi-hole DNS, Unbound DNS, fallback DNS, Uptime Kuma, and Docker root state. Default output shows only warnings and failures.
doctor-cluster [--all|--details|--verbose|--json]: Diagnose Kubernetes API, nodes, core pods, MetalLB, Traefik, app health, and Pimox worker tailnet egress. Default output shows only warnings and failures.
inventory-check: Validate homelab.yml with a real YAML parser when available.
secrets-check: Validate SOPS and age configuration plus committed encrypted secret file structure.
tailnet-policy-check: Validate the repo-managed Tailscale ACL policy.
ai-check: Check local AI helper prerequisites, knowledge index, and Ollama availability when the backstage helper is enabled.
ask QUESTION...: Search the local homelab knowledge index for relevant docs, runbooks, scripts, and commands. Set LAB_AI_ASK_LLM=true to ask Ollama after retrieval.
ask --citations QUESTION...: Return cited retrieval context with source paths and line ranges, without calling a model.
ai-evals {list|run|show}: Run deterministic Jeannie explanation and RAG grounding eval cases from infra/ai-evals. The harness is read-only and uses fixtures plus allowlisted commands so it can run before CI or homelab changes.
ai-scheduler {simulate|recommend NAME}: Simulate AI workload placement against repo-managed node and workload policy, including CPU, memory, architecture, tailnet access, and control-plane penalty.
ai-memory {check}: Check whether the local homelab RAG index manifest is present and whether indexed source files changed after indexing.
model-observe {prompts|run|report}: Track local model behavior against stable prompts. run --offline uses deterministic fixtures; run --live probes the configured Ollama endpoint.

Defensive Security

security-scan: Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP baseline passive scans for targets in security/targets.txt.
security-prepare: Validate Docker and report prerequisites, then pre-pull scanner images.
security-zap: Run only OWASP ZAP baseline passive scans against public targets.
security-k8s: Run kube-bench CIS checks against the Debian kubeadm host.
security-host: Run a Lynis host audit on Debian and print one highest-risk finding to focus on first. This command is report-only.
security-trivy: Run only Trivy filesystem and IaC/config scans.
security-secrets: Run only gitleaks secret scanning against this repo.
security-nuclei: Run low-rate nuclei HTTP exposure scans against public targets.
security-web: Run quick HTTPS, status, and defensive-header checks against owned web targets.
security-logs: Summarize recent Gitea and Traefik logs for common scanner paths, suspicious user agents, auth failures, and HTTP errors.
security-runtime: Check the Tetragon runtime-detection DaemonSet and print the command for watching recent events.
security-attack-path [namespace] [workload]: Run a read-only compromise-path drill for a Kubernetes workload. Defaults to the public website deployment.
prompt-injection-lab {list|run|show}: Run local prompt-injection defense drills from security/prompt-injection-lab. Cases cover hostile runbooks, tool output, fake dashboard mitigations, and unsafe command arguments.
red-blue {plan|run|ledger}: Plan or run the homelab red-team/blue-team loop. run --local executes only deterministic local checks; live security scans remain explicit.
Kyverno hardening policies: apps/supply-chain-policy includes audit-mode checks for privileged pods, privilege escalation, hostPath use, resource requests/limits, and mutable image tags.
apps/security-lab: GitOps-managed internal namespace for intentionally vulnerable practice targets. Use kubectl port-forward; do not expose it publicly.

Maintenance

state-backup: Create a local tarball backup of OpenTofu state and generated homelab state files.
fix-debian-docker-root: Repair or migrate the Debian Docker root according to inventory settings.
move-prometheus-stack-workers: Move Prometheus stack workloads off the control plane. Requires LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true because Prometheus PVC data is deleted.
secrets-init: Initialize local age and SOPS configuration on the Debian host.
ai-index: Build the local homelab knowledge index for the backstage AI helper.
ollama-setup: Install or configure Ollama on the Debian host and pull the configured local model.
website-translation-model: Prepare the website translation model assets.
website-ollama-listen: Configure the Ollama service listen address for website use.

COMMON ENVIRONMENT

HOMELAB_INVENTORY_FILE: Path to the canonical non-secret inventory. Defaults to ./homelab.yml.
LAB_STATUS_LOG_SINCE: Log window for status log checks. Defaults to 30m.
LAB_STATUS_RESTART_THRESHOLD: Pod restart count that turns the status restart-pressure check into a warning. Defaults to 3.
LAB_STATUS_DISK_USE_THRESHOLD: Disk usage percentage that turns the status disk check into a warning. Defaults to 85.
HOMELAB_STATE_DIR: Local runtime state directory. Defaults to ${XDG_DATA_HOME:-$HOME/.local/share}/homelab.
JEANNIE_LOG_DIR: Directory for command logs. Defaults to $HOMELAB_STATE_DIR/logs.
JEANNIE_VERBOSE=true: Print full step output as it runs. Without this, noisy commands are compact and full output is written to the log file.
JEANNIE_PROGRESS_INTERVAL_SECONDS: Refresh interval for compact long-running step progress lines. Defaults to 10. Compact steps show elapsed time and the latest sanitized log line while the full output continues writing to the log file.
JEANNIE_CORE_BIN: Optional path to a prebuilt Go jeannie-core binary. Defaults to .lab/bin/jeannie-core; wrappers build it on demand when Go is available.
LAB_STATUS_DETAILS=true: With status --verbose, append detailed host, Docker, Kubernetes, Pimox, RPi, Tailscale, and HTTP tables after the normal status cascade.
LAB_STATUS_SHOW_OK_OUTPUT=true: Show short command output for successful cascade checks. Failed and warning checks already show short output by default.
LAB_PIHOLE_STATUS_QUERY: DNS name used by status and doctor-preapply for the Pi-hole resolver smoke test. Defaults to cloudflare.com.
SECURITY_REPORT_DIR: Output directory for defensive security reports. Defaults to $HOMELAB_STATE_DIR/security-reports.
SECURITY_TARGETS_FILE: Public target allowlist for ZAP and nuclei scans. Defaults to security/targets.txt.
SECURITY_ZAP_TARGET: Optional single URL override for security-zap.
SECURITY_NUCLEI_TARGET: Optional single URL override for security-nuclei.
SECURITY_WEB_TARGET: Optional single URL override for security-web.
SECURITY_LOG_SINCE: Log window for security-logs. Defaults to 24h.
lynis: security-host requires Lynis to be installed on the Debian host. Install it explicitly with sudo apt install lynis.
LAB_BACKSTAGE_BRAIN_ENABLED=true: Allows security-host to ask the local Ollama/RAG helper to choose the single highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie falls back to the first Lynis warning, then the first suggestion.
LAB_AUTO_APPROVE=false: Disable automatic OpenTofu approval for apply paths that support confirmation.
LAB_PIMOX_WORKER_COUNT: Desired Pimox worker count when not supplied by generated cluster topology state.
LAB_PIMOX_WORKER_REQUIRE_LAN_PROBE: Require the worker guest to ping the LAN gateway and probe host after static IP assignment. Defaults to true.
LAB_PIMOX_WORKER_REQUIRE_HOST_PROBE: Require the Debian host to ping the worker IP before waiting for SSH. Defaults to true.
LAB_SKIP_PREFLIGHT=true: Skip preflight checks. Use only for targeted troubleshooting.
LAB_PREFLIGHT_ROOT_MIN_FREE_GIB: Minimum Debian / free space for doctor-preapply. Default is 10.
LAB_PREFLIGHT_DATA_MIN_FREE_GIB: Minimum Debian /data free space for doctor-preapply. Default is 20.
LAB_PREFLIGHT_EDGE_MIN_FREE_GIB: Minimum OCI edge / free space for doctor-preapply. Default is 2.
LAB_GITEA_INSTALL_DOCKER=true: Allow deploy-gitea to install Docker through get.docker.com when Docker is missing. Default is to fail and ask for host bootstrap.
LAB_RPI_INSTALL_DOCKER=true: Allow rpi-services to install Docker through get.docker.com on the Raspberry Pi when Docker is missing. Default is to fail.
TF_VAR_edge_install_docker=true: Allow the edge OpenTofu stack to install Docker on the OCI host when Docker is missing. Default is to fail.
TF_VAR_haproxy_stats_password: Required HAProxy stats password for the edge stack. It must be set explicitly and must not use the old default.
LAB_CONFIRM_NUKE=homelab: Required confirmation for direct ./jeannie nuke.
LAB_NUKE_DESTROY_PIMOX_WORKERS=true: Also destroy Pimox worker VMs during nuke.
LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true: Required confirmation for move-prometheus-stack-workers.

FILES

homelab.yml: Canonical non-secret inventory for hosts, IPs, domains, ports, and the main script name.
.lab/cluster-workers.auto.tfvars.json: Generated worker topology passed to the cluster stack.
.lab/pimox-workers.tsv: Generated Pimox worker inventory.
.lab/manual-workers.tsv: User-managed manual worker inventory.
$HOMELAB_STATE_DIR/logs: Jeannie command logs.
$HOMELAB_STATE_DIR/tofu-state-backups: Local OpenTofu and generated state backups.
$HOMELAB_STATE_DIR/security-reports: Defensive scan reports.
security/targets.txt: Public target allowlist for defensive web scans.

EXAMPLES

Start a three-worker homelab:

LAB_PIMOX_WORKER_COUNT=3 ./jeannie up

Preview all OpenTofu changes:

./jeannie plan

Run focused edge diagnostics:

./jeannie doctor-edge

Run the default defensive scan set:

./jeannie security-scan

Prepare scanner images before the first run:

./jeannie security-prepare

Run a ZAP baseline scan against one URL:

SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap

Run a report-only Lynis host audit:

./jeannie security-host

Run the blocking pre-apply doctor directly:

./jeannie doctor-preapply

Create a local state backup:

./jeannie state-backup

Destroy the cluster path intentionally:

LAB_CONFIRM_NUKE=homelab ./jeannie nuke

EXIT STATUS

0: Command completed successfully.
Non-zero: A check, deployment step, or external tool failed. Check the printed error and the referenced Jeannie log file for full command output.

23 KiB Raw Blame History