16 KiB

Raw Blame History

JEANNIE(1)

NAME

jeannie - operate the personal homelab control pipeline

SYNOPSIS

./jeannie COMMAND [ARG...]

DESCRIPTION

jeannie is the main homelab operator script. It deploys and checks the Debian control host, Gitea, Raspberry Pi services, Pimox workers, Kubernetes platform components, applications, public edge routing, secrets tooling, and local AI helpers.

Run mutating infrastructure commands from the Debian homelab host. Commands that can destroy or rewrite state have explicit confirmation environment variables. The canonical script name is metadata.main_script in homelab.yml.

COMMANDS

Core Lifecycle

up: Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi services, runs full preflight and pre-apply doctor checks, then applies Pimox, OpenWrt, cluster, platform, apps, and edge stacks.
plan [all|provisioning|cluster|platform|apps|edge]: Run OpenTofu plans without applying changes. The default target is all.
rebuild-cluster: Back up current local OpenTofu state, destroy cluster state through the guarded nuke path, rebuild Pimox workers, recreate the cluster, and redeploy platform, apps, and edge.
stop-cluster: Stop the Kubernetes control-plane runtime and worker VMs without destroying cluster state.
start-cluster: Start the Kubernetes control-plane runtime and desired worker VMs.
status: Print a cascade health report for the configured homelab, from bootstrap readiness through public website health. Set LAB_STATUS_DETAILS=true to append the older detailed tables. The cascade includes "what broke" signals for recent deployment readiness, pod restarts, node pressure, disk pressure, Traefik 5xx/404 log evidence, and recent Gitea errors.
capacity: Print a read-only placement report covering Debian memory/disk/Docker usage, Kubernetes node and PVC usage, Pimox VM/storage allocation, RPi Docker/disk state, and practical placement guidance.
recover-plan: Print the ordered disaster recovery checklist and lightweight prerequisite checks for Debian, Gitea, RPi DNS, Pimox, Kubernetes, GitOps apps, and edge verification.
recover-power [--dry-run]: Run the post-outage recovery sequence in dependency order: Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and edge/public checks.
scorecard: Roll up backup readiness, GitOps health, DNS/RPi health, cluster health, edge health, capacity pressure, security posture, public certificates, inventory, and docs freshness into a compact pass/warn/fail report.
Report renderer: Report-oriented commands can share scripts/report-render, which reads status TSV rows and prints compact grouped summaries, detailed output, or JSON.
gitops-status: Print a focused Argo CD status view with application sync/health, out-of-sync or degraded apps, repository secret presence, recent events, and recent repo-server/application-controller errors.
cert-check: Verify public DNS, TLS certificate expiry, public website/Gitea HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory.
release-snapshot: Write a timestamped pre-change report with Git state, Kubernetes/Argo CD/Helm state, workload images, public URL status, and latest Gitea/OpenTofu backup pointers.
backup-status: Report freshness for Gitea backups, OpenTofu state backups, restore drill reports, and repo-managed Pi-hole restore inputs. The main status cascade includes this as a warning signal.
synthetic-checks: Run end-to-end probes for Gitea HTTP/SSH, registry API, Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. Registry push/pull is opt-in through LAB_SYNTHETIC_REGISTRY_PUSH=true.
resource-budget: Check the report-only resource policy in infra/resource-budgets.yml against Kubernetes pod requests/limits and Debian disk/memory signals.
artifact-cache {status|up|down|instructions}: Manage the optional Debian-host artifact cache stack for apt packages and Docker Hub pull-through image caching.
golden-ledger {show|check}: Show or validate the Pimox golden image version ledger.
route-inventory: Report Kubernetes ingress hosts, paths, backend services, TLS coverage, and visible Uptime Kuma monitor coverage.
workers <command>: Manage Kubernetes/Pimox worker lifecycle operations. Commands include list, start, stop, drain, uncordon, recreate-plan, and rebalance.
change-journal {list|path}: List local journal entries written before risky commands, or print the journal directory path.
map: Print a dependency map from the canonical inventory. Use map --dot for a Graphviz DOT graph.
Prometheus alerts: High-signal alert rules are managed as the apps/homelab-alerts GitOps app. They cover node readiness, restart storms, unavailable deployments, storage pressure, Traefik 5xx, and core service visibility.
NetworkPolicies: Owned application namespaces include egress policies. The website can reach DNS, Redis, and the Debian Ollama endpoint; the security lab can reach DNS and external web targets while remaining isolated from cluster and LAN ranges.
validate: Run non-mutating repository checks used by the GitOps promotion flow: inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu formatting, tailnet policy validation, and optional static security checks.
access-audit: Run a read-only access inventory across SSH targets, Git/Gitea remotes, Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the Gitea runner.
kubeconfig-readonly: Generate a separate local read-only kubeconfig from the repo-managed homelab-access/homelab-readonly ServiceAccount.
nuke: Destroy Kubernetes state and Pimox worker VMs. Requires LAB_CONFIRM_NUKE=homelab.

Deploy Targets

apps: Deploy platform applications and build or publish application images when needed.
deploy-gitea: Deploy the external Gitea Docker Compose service on the Debian host.
rpi-services: Deploy Pi-hole, Unbound, and Uptime Kuma on the Raspberry Pi.
bootstrap-gitea-repo: Ensure the configured Gitea repository, branch, remotes, and host SSH key are ready.
openwrt: Apply the OpenWrt VM stack.

Gitea Operations

backup-gitea: Run a Gitea backup on the Debian host.
drill-restore: Run a read-only restore checklist for Gitea backups, Pi-hole repo-managed config, OpenTofu state backup archives, and GitOps rebuild inputs.
drill-gitea-restore: Run the Gitea restore drill workflow.
drill-pihole-restore: Validate that repo-managed Pi-hole restore inputs exist and are non-empty.
install-gitea-runner [TOKEN]: Install or repair the Gitea Actions runner. If TOKEN is omitted, the command uses the configured token source.

Checks And Doctors

preflight: Run homelab preflight checks. This verifies inventory-derived host settings, Docker roots, Tailscale IPs, Gitea reachability, Pimox storage, and OCI edge SSH.
doctor-preapply: Run blocking readiness checks used by up before high-blast-radius applies. It checks Debian free disk, Docker and containerd, Raspberry Pi Docker writability, Pimox storage, OCI edge disk, and Pi-hole DNS.
doctor-versions: Check Kubernetes component version alignment.
doctor-edge: Diagnose public edge routing, Traefik reachability, and Gitea edge exposure.
doctor-gitea: Diagnose Debian Gitea container, local access, edge access, and repository health.
doctor-rpi: Diagnose Raspberry Pi services, Pi-hole DNS, Unbound DNS, fallback DNS, Uptime Kuma, and Docker root state.
doctor-cluster: Diagnose Kubernetes API, nodes, core pods, MetalLB, Traefik, and app health.
inventory-check: Validate homelab.yml with a real YAML parser when available.
secrets-check: Validate SOPS and age configuration plus committed encrypted secret file structure.
tailnet-policy-check: Validate the repo-managed Tailscale ACL policy.
ai-check: Check local AI helper prerequisites, knowledge index, and Ollama availability when the backstage helper is enabled.

Defensive Security

security-scan: Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP baseline passive scans for targets in security/targets.txt.
security-prepare: Validate Docker and report prerequisites, then pre-pull scanner images.
security-zap: Run only OWASP ZAP baseline passive scans against public targets.
security-k8s: Run kube-bench CIS checks against the Debian kubeadm host.
security-host: Run a Lynis host audit on Debian and print one highest-risk finding to focus on first. This command is report-only.
security-trivy: Run only Trivy filesystem and IaC/config scans.
security-secrets: Run only gitleaks secret scanning against this repo.
security-nuclei: Run low-rate nuclei HTTP exposure scans against public targets.
security-web: Run quick HTTPS, status, and defensive-header checks against owned web targets.
security-logs: Summarize recent Gitea and Traefik logs for common scanner paths, suspicious user agents, auth failures, and HTTP errors.
security-runtime: Check the Tetragon runtime-detection DaemonSet and print the command for watching recent events.
security-attack-path [namespace] [workload]: Run a read-only compromise-path drill for a Kubernetes workload. Defaults to the public website deployment.
Kyverno hardening policies: apps/supply-chain-policy includes audit-mode checks for privileged pods, privilege escalation, hostPath use, resource requests/limits, and mutable image tags.
apps/security-lab: GitOps-managed internal namespace for intentionally vulnerable practice targets. Use kubectl port-forward; do not expose it publicly.

Maintenance

state-backup: Create a local tarball backup of OpenTofu state and generated homelab state files.
fix-debian-docker-root: Repair or migrate the Debian Docker root according to inventory settings.
move-prometheus-stack-workers: Move Prometheus stack workloads off the control plane. Requires LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true because Prometheus PVC data is deleted.
secrets-init: Initialize local age and SOPS configuration on the Debian host.
ai-index: Build the local homelab knowledge index for the backstage AI helper.
ollama-setup: Install or configure Ollama on the Debian host and pull the configured local model.
website-translation-model: Prepare the website translation model assets.
website-ollama-listen: Configure the Ollama service listen address for website use.

COMMON ENVIRONMENT

HOMELAB_INVENTORY_FILE: Path to the canonical non-secret inventory. Defaults to ./homelab.yml.
LAB_STATUS_LOG_SINCE: Log window for status log checks. Defaults to 30m.
LAB_STATUS_RESTART_THRESHOLD: Pod restart count that turns the status restart-pressure check into a warning. Defaults to 3.
LAB_STATUS_DISK_USE_THRESHOLD: Disk usage percentage that turns the status disk check into a warning. Defaults to 85.
HOMELAB_STATE_DIR: Local runtime state directory. Defaults to ${XDG_DATA_HOME:-$HOME/.local/share}/homelab.
JEANNIE_LOG_DIR: Directory for command logs. Defaults to $HOMELAB_STATE_DIR/logs.
JEANNIE_VERBOSE=true: Print full step output as it runs. Without this, noisy commands are compact and full output is written to the log file.
LAB_STATUS_DETAILS=true: Append detailed host, Docker, Kubernetes, Pimox, RPi, Tailscale, and HTTP tables after the normal status cascade.
LAB_STATUS_SHOW_OK_OUTPUT=true: Show short command output for successful cascade checks. Failed and warning checks already show short output by default.
LAB_PIHOLE_STATUS_QUERY: DNS name used by status and doctor-preapply for the Pi-hole resolver smoke test. Defaults to cloudflare.com.
SECURITY_REPORT_DIR: Output directory for defensive security reports. Defaults to $HOMELAB_STATE_DIR/security-reports.
SECURITY_TARGETS_FILE: Public target allowlist for ZAP and nuclei scans. Defaults to security/targets.txt.
SECURITY_ZAP_TARGET: Optional single URL override for security-zap.
SECURITY_NUCLEI_TARGET: Optional single URL override for security-nuclei.
SECURITY_WEB_TARGET: Optional single URL override for security-web.
SECURITY_LOG_SINCE: Log window for security-logs. Defaults to 24h.
lynis: security-host requires Lynis to be installed on the Debian host. Install it explicitly with sudo apt install lynis.
LAB_BACKSTAGE_BRAIN_ENABLED=true: Allows security-host to ask the local Ollama/RAG helper to choose the single highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie falls back to the first Lynis warning, then the first suggestion.
LAB_AUTO_APPROVE=false: Disable automatic OpenTofu approval for apply paths that support confirmation.
LAB_PIMOX_WORKER_COUNT: Desired Pimox worker count when not supplied by generated cluster topology state.
LAB_SKIP_PREFLIGHT=true: Skip preflight checks. Use only for targeted troubleshooting.
LAB_PREFLIGHT_ROOT_MIN_FREE_GIB: Minimum Debian / free space for doctor-preapply. Default is 10.
LAB_PREFLIGHT_DATA_MIN_FREE_GIB: Minimum Debian /data free space for doctor-preapply. Default is 20.
LAB_PREFLIGHT_EDGE_MIN_FREE_GIB: Minimum OCI edge / free space for doctor-preapply. Default is 2.
LAB_GITEA_INSTALL_DOCKER=true: Allow deploy-gitea to install Docker through get.docker.com when Docker is missing. Default is to fail and ask for host bootstrap.
LAB_RPI_INSTALL_DOCKER=true: Allow rpi-services to install Docker through get.docker.com on the Raspberry Pi when Docker is missing. Default is to fail.
TF_VAR_edge_install_docker=true: Allow the edge OpenTofu stack to install Docker on the OCI host when Docker is missing. Default is to fail.
TF_VAR_haproxy_stats_password: Required HAProxy stats password for the edge stack. It must be set explicitly and must not use the old default.
LAB_CONFIRM_NUKE=homelab: Required confirmation for direct ./jeannie nuke.
LAB_NUKE_DESTROY_PIMOX_WORKERS=true: Also destroy Pimox worker VMs during nuke.
LAB_CONFIRM_DELETE_PROMETHEUS_DATA=true: Required confirmation for move-prometheus-stack-workers.

FILES

homelab.yml: Canonical non-secret inventory for hosts, IPs, domains, ports, and the main script name.
.lab/cluster-workers.auto.tfvars.json: Generated worker topology passed to the cluster stack.
.lab/pimox-workers.tsv: Generated Pimox worker inventory.
.lab/manual-workers.tsv: User-managed manual worker inventory.
$HOMELAB_STATE_DIR/logs: Jeannie command logs.
$HOMELAB_STATE_DIR/tofu-state-backups: Local OpenTofu and generated state backups.
$HOMELAB_STATE_DIR/security-reports: Defensive scan reports.
security/targets.txt: Public target allowlist for defensive web scans.

EXAMPLES

Start a three-worker homelab:

LAB_PIMOX_WORKER_COUNT=3 ./jeannie up

Preview all OpenTofu changes:

./jeannie plan

Run focused edge diagnostics:

./jeannie doctor-edge

Run the default defensive scan set:

./jeannie security-scan

Prepare scanner images before the first run:

./jeannie security-prepare

Run a ZAP baseline scan against one URL:

SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap

Run a report-only Lynis host audit:

./jeannie security-host

Run the blocking pre-apply doctor directly:

./jeannie doctor-preapply

Create a local state backup:

./jeannie state-backup

Destroy the cluster path intentionally:

LAB_CONFIRM_NUKE=homelab ./jeannie nuke

EXIT STATUS

0: Command completed successfully.
Non-zero: A check, deployment step, or external tool failed. Check the printed error and the referenced Jeannie log file for full command output.

16 KiB Raw Blame History