Add Jeannie status command

README.md

@@ -28,7 +28,7 @@ The lab is intentionally small but production-shaped:
   Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream
   keepalive
 
-Run `./lab.sh up` and `./lab.sh nuke` only from the Debian homelab server. The
+Run `./jeannie up` and `./jeannie nuke` only from the Debian homelab server. The
 script intentionally refuses to run from non-Debian machines so a laptop cannot
 accidentally modify the cluster.
 
@@ -39,7 +39,7 @@ accidentally modify the cluster.
    - serves Debian 13 arm64 netboot assets through TFTP and HTTP
    - creates a golden image install path with Kubernetes, containerd,
      qemu-guest-agent, cloud-init, and storage client packages ready
-   - is driven by `./lab.sh up` when Pimox is reachable, without changing
+   - is driven by `./jeannie up` when Pimox is reachable, without changing
      Orange Pi host networking
 
 2. `bootstrap/cluster`
@@ -122,14 +122,14 @@ plane upgrade.
 Check node versions from the Debian host:
 
 ```bash
-./lab.sh doctor-versions
+./jeannie doctor-versions
 ```
 
 The check prints the API server, kubelet versions, kubelet minors, and containerd
 runtimes. It exits nonzero when kubelet minors differ from the API server minor
 or when containerd major versions are mixed. The Gitea Actions auto path runs
 this check before app deploys; kubelet minor drift switches the job to
-`./lab.sh rebuild-cluster` so the Pimox template and worker VMs are replaced
+`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced
 from the pinned image. For Pimox workers, prefer rebuilding from the corrected
 golden template and rejoining the worker over ad hoc live package upgrades.
 
@@ -139,7 +139,7 @@ From the Debian server:
 
 ```bash
 cd ~/my-homelab-configs
-./lab.sh up
+./jeannie up
 ```
 
 The script first deploys external Gitea to the Debian host with Docker Compose
@@ -170,7 +170,7 @@ must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID
 `9010` and VMID `9011`.
 
 OpenWrt firewall VM automation is available as a standalone command because it
-attaches to both WAN and LAN bridges. Run `./lab.sh openwrt` after `vmbr1`
+attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1`
 already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM
 SystemReady EFI image, writes basic WAN/LAN/firewall config into the image,
 imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores
@@ -226,9 +226,19 @@ The website should be reached through the configured public hostname, not the ra
 OCI IP address, because the Let's Encrypt certificate is issued for the
 hostname.
 
+Run a read-only health snapshot from the Debian server with:
+
+```bash
+./jeannie status
+```
+
+It reports host memory/disk, systemd services, Docker Compose stacks,
+Kubernetes health when the API is reachable, Pimox worker VM status, Tailscale,
+and key local/public HTTP endpoints.
+
 ## Adding Nodes
 
-For Pimox on Orange Pi 5 Plus, `./lab.sh up` can create the Debian 13 arm64
+For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64
 template and worker VM clones automatically when the Orange Pi storage path is
 healthy. Defaults are intentionally tied to the observed host: Pimox SSH host
 `192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two
@@ -248,7 +258,7 @@ allowing higher indexes to be automated.
 Run a full cluster rebuild from the Debian server with:
 
 ```bash
-./lab.sh rebuild-cluster
+./jeannie rebuild-cluster
 ```
 
 That path preserves external Gitea, rebuilds the Pimox template
@@ -262,7 +272,7 @@ To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state,
 PV data, or VM disks, run:
 
 ```bash
-./lab.sh stop-cluster
+./jeannie stop-cluster
 ```
 
 This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when
@@ -278,7 +288,7 @@ hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`,
 select VM workers. Resume the runtime with:
 
 ```bash
-./lab.sh start-cluster
+./jeannie start-cluster
 ```
 
 To exclude the Raspberry Pi from the Kubernetes cluster, set
@@ -328,7 +338,7 @@ preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and
 kube-state-metrics scheduling does not collapse onto the first worker that joins.
 PVC-backed monitoring StatefulSets are intentionally treated separately because
 their retained OpenEBS hostpath volumes are node-local. Run
-`./lab.sh move-prometheus-stack-workers` from the Debian host to label existing
+`./jeannie move-prometheus-stack-workers` from the Debian host to label existing
 worker nodes, destroy only the existing `prometheus-stack` Helm release, delete
 its retained PVC/PV objects, and recreate the stack on the worker selector when
 you intentionally accept losing that monitoring data. A planned monitoring data
@@ -395,7 +405,7 @@ The edge stack uses Traefik as its backend by default and validates
 subnet-route approval is not automatic in the tailnet policy, the edge deploy
 will fail clearly instead of silently keeping a broken public path.
 
-For `./lab.sh nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of
+For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of
 remote SSH targets when more worker nodes exist. Set it to an empty string for a
 single-node rebuild.
 
@@ -423,13 +433,13 @@ and audits whether the image has both:
 The private Cosign key, generated password, and downloaded Cosign binary live in
 `${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing
 identity survives temporary checkout cleanup. Repo-local image state files still
-live under `.lab/`, which is ignored by Git. `./lab.sh apps` creates or reuses
+live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses
 the persistent Cosign key, publishes the public key into the Kyverno namespace as
 `homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs
 the pushed images, attaches a signed SPDX SBOM predicate, and verifies both
 artifacts before refreshing the workload Applications. The policy starts in
 `Audit` mode so an existing live deployment can recover while the current images
-are signed; after `./lab.sh apps` verifies both images, change `validationActions` in
+are signed; after `./jeannie apps` verifies both images, change `validationActions` in
 `apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to
 `Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to
 manage the key password externally; otherwise the Debian runner creates
@@ -513,7 +523,7 @@ certificate warning because the trusted certificate is issued for the hostname.
 The edge stack uses HTTP-01 validation and requests one certificate covering
 `server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains
 under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must
-be open before `./lab.sh up` runs. Set `TF_VAR_letsencrypt_email` to receive
+be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive
 expiry notices, or leave it empty to register without an email. Set
 `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate.
 
@@ -523,7 +533,7 @@ Add Kubernetes manifests under `apps/<name>` and register them in
 `bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and
 self-healing for the app.
 
-The `heimdall` app is intentionally waited on at the end of `./lab.sh apps`.
+The `heimdall` app is intentionally waited on at the end of `./jeannie apps`.
 It runs the LinuxServer.io Heimdall dashboard, persists `/config` on
 OpenEBS, and seeds tiles for the website, demo apps, Gitea, Grafana,
 Prometheus, Alertmanager, Argo CD, the local registry, and Heimdall itself.
@@ -598,12 +608,12 @@ restart `argocd-repo-server`, and hard-refresh the affected Application.
 Deploy or refresh the external Gitea container from the Debian host with:
 
 ```bash
-./lab.sh deploy-gitea
+./jeannie deploy-gitea
 ```
 
 ## Gitea Backups
 
-`./lab.sh up` installs a Debian-host systemd timer named
+`./jeannie up` installs a Debian-host systemd timer named
 `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured
 Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the
 dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default
@@ -618,13 +628,13 @@ does not write into the live Gitea data directory.
 Run a manual backup from the Debian server with:
 
 ```bash
-./lab.sh backup-gitea
+./jeannie backup-gitea
 ```
 
 Run the restore drill manually with:
 
 ```bash
-./lab.sh drill-gitea-restore
+./jeannie drill-gitea-restore
 ```
 
 Useful checks:
@@ -646,16 +656,16 @@ a repository-scoped Debian host runner with the label `homelab-debian`.
 The workflow only blocks automatic deploy for external Gitea service
 changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
 `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
-functions in `lab.sh`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
-default: Actions runs `./lab.sh doctor-versions` when the Debian runner already
-has a cluster kubeconfig. If node versions are aligned it runs `./lab.sh apps`;
-if kubelet minor drift is detected it runs `./lab.sh rebuild-cluster`; if no
-kubeconfig exists it also runs `./lab.sh rebuild-cluster`.
+functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
+default: Actions runs `./jeannie doctor-versions` when the Debian runner already
+has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`;
+if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no
+kubeconfig exists it also runs `./jeannie rebuild-cluster`.
 
 Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
 on the runner to force one path.
 
-`./lab.sh bootstrap-gitea-repo` also registers the Debian host SSH public key
+`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key
 with the Gitea repository and switches the Debian working copy's `gitea` remote
 to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key
 is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
@@ -677,7 +687,7 @@ Register and start the Debian runner from the Debian server:
 
 ```bash
 cd ~/my-homelab-configs
-GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./lab.sh install-gitea-runner
+GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./jeannie install-gitea-runner
 ```
 
 The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
@@ -700,12 +710,12 @@ journalctl -u homelab-gitea-runner.service -n 100 --no-pager
 providers, Helm chart versions, and the pinned tools used by the Gitea Actions
 workflow. Renovate should open reviewable update branches or PRs only; it must
 not auto-merge infrastructure changes. Keep app-only dependency updates on the
-normal Gitea Actions path, and run `./lab.sh up` manually on the Debian server
+normal Gitea Actions path, and run `./jeannie up` manually on the Debian server
 for platform or provisioning updates.
 
 ## Destructive Rebuilds
 
-`./lab.sh nuke` resets kubeadm, containerd runtime state, CNI files, Calico
+`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico
 links, iptables rules, and local OpenTofu state. It does not delete retained data
 under `/data/openebs/local`.
 
@@ -735,14 +745,14 @@ Create or refresh the Ollama model on the Debian server before deploying a
 website image that points at it:
 
 ```bash
-./lab.sh website-translation-model
+./jeannie website-translation-model
 ```
 
 Ollama must also listen on the Debian host LAN address so Kubernetes pods on
 other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`:
 
 ```bash
-./lab.sh website-ollama-listen
+./jeannie website-ollama-listen
 ```
 
 The CV page has two client-side presentation modes:
@@ -757,22 +767,22 @@ served from a separate `demos-static` artifact under `apps/demos-static` and are
 published through the `demos-static` Argo CD application. Public traffic reaches
 them through the edge path at `/demo-apps/`.
 
-`./lab.sh up` builds and pushes two independent images:
+`./jeannie up` builds and pushes two independent images:
 
-- a content-hash `php-website` tag generated by `lab.sh` and passed to Argo CD
+- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD
   as a Kustomize image override
 - `demos-static:latest` from `apps/demos-static`
 - Cosign signatures and signed SPDX SBOM attestations for both pushed images
 
 The website manifest keeps the stable base image name `php-website:bootstrap`.
-During bootstrap, `lab.sh` hashes `apps/website`, builds
+During bootstrap, `jeannie` hashes `apps/website`, builds
 `<registry>/php-website:src-<hash>`, exports that exact reference through
 `TF_VAR_website_image_ref`, and the Argo CD Application applies it through
 Kustomize. This keeps the GitOps source generic while the deployed image remains
 immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
 image digest, so the workload runs the same digest that was signed and attested.
 
-After `./lab.sh apps`, the live deployment image should be a content-hash tag,
+After `./jeannie apps`, the live deployment image should be a content-hash tag,
 for example `192.168.100.73:30500/php-website:src-...`. If it still shows
 `php-website:latest`, Argo CD has not rendered the current Application source.
 Check the `website-production` Application source, sync status, and repository

jeannie

@@ -1925,7 +1925,7 @@ if not isinstance(document, dict):
         "documentNamespace": f"https://homelab.local/spdx/{safe_ref}/{source_hash}",
         "creationInfo": {
             "created": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
-            "creators": ["Tool: lab.sh"],
+            "creators": ["Tool: jeannie"],
         },
         "packages": [
             {
@@ -3430,6 +3430,180 @@ start_cluster() {
     echo "Kubernetes runtime start requested. Use 'kubectl get nodes -o wide' to watch readiness."
 }
 
+status_section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+status_run() {
+    local description="$1"
+    shift
+
+    printf '\n-- %s\n' "${description}"
+    if ! "$@"; then
+        printf 'status check failed: %s\n' "${description}" >&2
+    fi
+}
+
+status_http() {
+    local name="$1"
+    local url="$2"
+
+    if command -v curl >/dev/null 2>&1; then
+        printf '%-28s ' "${name}"
+        curl -k -sS -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code} %{url_effective}\n' "${url}" ||
+            printf 'unreachable %s\n' "${url}"
+    else
+        printf '%-28s curl not installed\n' "${name}"
+    fi
+}
+
+status_systemd_units() {
+    local unit
+
+    for unit in "$@"; do
+        if systemctl list-unit-files --no-legend "${unit}.service" 2>/dev/null | grep -q .; then
+            printf '%-28s %s\n' "${unit}" "$(systemctl is-active "${unit}" 2>/dev/null || true)"
+        else
+            printf '%-28s not-installed\n' "${unit}"
+        fi
+    done
+}
+
+status_compose_stack() {
+    local name="$1"
+    local directory="$2"
+
+    printf '\n-- Docker Compose: %s (%s)\n' "${name}" "${directory}"
+    if [[ ! -d "${directory}" ]]; then
+        printf 'missing directory\n'
+        return 0
+    fi
+    if ! command -v docker >/dev/null 2>&1; then
+        printf 'docker not installed\n'
+        return 0
+    fi
+    if ! sudo docker compose version >/dev/null 2>&1; then
+        printf 'docker compose unavailable\n'
+        return 0
+    fi
+
+    (cd "${directory}" && sudo docker compose ps) || true
+}
+
+status_kubernetes() {
+    status_section "Kubernetes"
+
+    if ! command -v kubectl >/dev/null 2>&1; then
+        printf 'kubectl not installed\n'
+        return 0
+    fi
+    if [[ ! -s "${KUBECONFIG_PATH}" ]]; then
+        printf 'kubeconfig missing: %s\n' "${KUBECONFIG_PATH}"
+        return 0
+    fi
+    if ! kubectl --kubeconfig "${KUBECONFIG_PATH}" get --raw=/readyz >/dev/null 2>&1; then
+        printf 'API server not reachable through %s\n' "${KUBECONFIG_PATH}"
+        return 0
+    fi
+
+    status_run "nodes" kubectl --kubeconfig "${KUBECONFIG_PATH}" get nodes -o wide
+    status_kubernetes_problem_pods
+    status_run "traefik services" kubectl --kubeconfig "${KUBECONFIG_PATH}" -n traefik get svc,pods -o wide
+    status_run "website pods" kubectl --kubeconfig "${KUBECONFIG_PATH}" -n website-production get pods -o wide
+}
+
+status_kubernetes_problem_pods() {
+    local rows
+
+    printf '\n-- pods not Running/Completed\n'
+    rows="$(
+        kubectl --kubeconfig "${KUBECONFIG_PATH}" get pods -A --no-headers -o wide |
+            awk '$4 != "Running" && $4 != "Completed" { print }'
+    )"
+    if [[ -n "${rows}" ]]; then
+        printf '%s\n' "${rows}"
+    else
+        printf 'none\n'
+    fi
+}
+
+status_pimox_workers() {
+    local pimox_host="${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}"
+    local pimox_user="${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}"
+    local pimox_key="${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-/home/jv/.ssh/id_ed25519}}"
+    local qm_bin="${LAB_PIMOX_QM_BIN:-${TF_VAR_pimox_qm_bin:-/usr/sbin/qm}}"
+    local worker_count="${LAB_PIMOX_WORKER_COUNT:-$(pimox_worker_count_default)}"
+    local worker_base_vmid="${LAB_PIMOX_WORKER_BASE_VMID:-9010}"
+    local worker_skip_indexes="${LAB_PIMOX_SKIP_WORKER_INDEXES:-}"
+    local index
+    local vmid
+
+    status_section "Pimox Workers"
+
+    if ! [[ "${worker_count}" =~ ^[0-9]+$ ]]; then
+        printf 'invalid LAB_PIMOX_WORKER_COUNT: %s\n' "${worker_count}"
+        return 0
+    fi
+    if ((worker_count == 0)); then
+        printf 'worker count is 0\n'
+        return 0
+    fi
+
+    for ((index = 1; index <= worker_count; index++)); do
+        if worker_index_is_skipped "${index}" "${worker_skip_indexes}"; then
+            printf 'worker index %s skipped\n' "${index}"
+            continue
+        fi
+        vmid=$((worker_base_vmid + index - 1))
+        printf 'VM %-6s ' "${vmid}"
+        pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo '${qm_bin}' status '${vmid}'" 2>/dev/null ||
+            printf 'unreachable or missing\n'
+    done
+}
+
+status_report() {
+    require_debian_server "status"
+
+    status_section "Host"
+    printf 'hostname: %s\n' "$(hostname -f 2>/dev/null || hostname)"
+    printf 'date:     %s\n' "$(date -Is)"
+    printf 'uptime:   %s\n' "$(uptime -p 2>/dev/null || uptime)"
+    status_run "memory" free -h
+    status_run "disk" df -h / /data /data/docker /data/openebs/local
+
+    status_section "Systemd"
+    status_systemd_units ssh docker containerd kubelet tailscaled homelab-gitea-runner
+
+    status_section "Docker"
+    if command -v docker >/dev/null 2>&1; then
+        status_run "docker containers" sudo docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
+    else
+        printf 'docker not installed\n'
+    fi
+    status_compose_stack "gitea" "/data/homelab-gitea"
+    status_compose_stack "arr-stack" "${REPO_ROOT}/infra/arr-stack"
+
+    status_kubernetes
+    status_pimox_workers
+
+    status_section "Tailscale"
+    if command -v tailscale >/dev/null 2>&1; then
+        status_run "tailscale ip" tailscale ip -4
+        status_run "tailscale peers" tailscale status
+    else
+        printf 'tailscale not installed\n'
+    fi
+
+    status_section "HTTP"
+    status_http "website public" "https://lab2025.duckdns.org/"
+    status_http "gitea public" "https://lab2025.duckdns.org/git/"
+    status_http "gitea local" "http://127.0.0.1:3000/"
+    status_http "traefik lb" "http://192.168.100.240/"
+    status_http "arr radarr" "http://127.0.0.1:7878/"
+    status_http "arr sonarr" "http://127.0.0.1:8989/"
+    status_http "arr prowlarr" "http://127.0.0.1:9696/"
+}
+
 nuke() {
     local worker_ssh_targets
     local worker_targets
@@ -3584,6 +3758,9 @@ case "${1:-}" in
     start-cluster)
         start_cluster
         ;;
+    status)
+        status_report
+        ;;
     apps)
         apps
         ;;
@@ -3621,7 +3798,7 @@ case "${1:-}" in
         nuke
         ;;
     *)
-        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|apps|website-translation-model|website-ollama-listen|deploy-gitea|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|openwrt|nuke}"
+        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|deploy-gitea|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|openwrt|nuke}"
         exit 1
         ;;
 esac