From 806efb8f850690fc2d7553eacfce0477b2cda22e Mon Sep 17 00:00:00 2001 From: juvdiaz Date: Sat, 27 Jun 2026 15:38:32 -0600 Subject: [PATCH] Add Jeannie status command --- README.md | 80 +++++++++++++----------- jeannie | 181 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 224 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 6063ca8..0a40326 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ The lab is intentionally small but production-shaped: Tailscale, with nginx caching, gzip, HTTP/2, TLS session reuse, and upstream keepalive -Run `./lab.sh up` and `./lab.sh nuke` only from the Debian homelab server. The +Run `./jeannie up` and `./jeannie nuke` only from the Debian homelab server. The script intentionally refuses to run from non-Debian machines so a laptop cannot accidentally modify the cluster. @@ -39,7 +39,7 @@ accidentally modify the cluster. - serves Debian 13 arm64 netboot assets through TFTP and HTTP - creates a golden image install path with Kubernetes, containerd, qemu-guest-agent, cloud-init, and storage client packages ready - - is driven by `./lab.sh up` when Pimox is reachable, without changing + - is driven by `./jeannie up` when Pimox is reachable, without changing Orange Pi host networking 2. `bootstrap/cluster` @@ -122,14 +122,14 @@ plane upgrade. Check node versions from the Debian host: ```bash -./lab.sh doctor-versions +./jeannie doctor-versions ``` The check prints the API server, kubelet versions, kubelet minors, and containerd runtimes. It exits nonzero when kubelet minors differ from the API server minor or when containerd major versions are mixed. The Gitea Actions auto path runs this check before app deploys; kubelet minor drift switches the job to -`./lab.sh rebuild-cluster` so the Pimox template and worker VMs are replaced +`./jeannie rebuild-cluster` so the Pimox template and worker VMs are replaced from the pinned image. For Pimox workers, prefer rebuilding from the corrected golden template and rejoining the worker over ad hoc live package upgrades. @@ -139,7 +139,7 @@ From the Debian server: ```bash cd ~/my-homelab-configs -./lab.sh up +./jeannie up ``` The script first deploys external Gitea to the Debian host with Docker Compose @@ -170,7 +170,7 @@ must be left untouched, or set `LAB_PIMOX_WORKER_COUNT=2` to manage both VMID `9010` and VMID `9011`. OpenWrt firewall VM automation is available as a standalone command because it -attaches to both WAN and LAN bridges. Run `./lab.sh openwrt` after `vmbr1` +attaches to both WAN and LAN bridges. Run `./jeannie openwrt` after `vmbr1` already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM SystemReady EFI image, writes basic WAN/LAN/firewall config into the image, imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores @@ -226,9 +226,19 @@ The website should be reached through the configured public hostname, not the ra OCI IP address, because the Let's Encrypt certificate is issued for the hostname. +Run a read-only health snapshot from the Debian server with: + +```bash +./jeannie status +``` + +It reports host memory/disk, systemd services, Docker Compose stacks, +Kubernetes health when the API is reachable, Pimox worker VM status, Tailscale, +and key local/public HTTP endpoints. + ## Adding Nodes -For Pimox on Orange Pi 5 Plus, `./lab.sh up` can create the Debian 13 arm64 +For Pimox on Orange Pi 5 Plus, `./jeannie up` can create the Debian 13 arm64 template and worker VM clones automatically when the Orange Pi storage path is healthy. Defaults are intentionally tied to the observed host: Pimox SSH host `192.168.100.80`, bridge `vmbr0`, template VMID `9000` on `local` storage, two @@ -248,7 +258,7 @@ allowing higher indexes to be automated. Run a full cluster rebuild from the Debian server with: ```bash -./lab.sh rebuild-cluster +./jeannie rebuild-cluster ``` That path preserves external Gitea, rebuilds the Pimox template @@ -262,7 +272,7 @@ To pause the Kubernetes cluster without deleting kubeadm files, OpenTofu state, PV data, or VM disks, run: ```bash -./lab.sh stop-cluster +./jeannie stop-cluster ``` This stops `kubelet`, stops Kubernetes CRI containers with `crictl` when @@ -278,7 +288,7 @@ hard `qm stop` after timeout, and the usual `LAB_PIMOX_WORKER_COUNT`, select VM workers. Resume the runtime with: ```bash -./lab.sh start-cluster +./jeannie start-cluster ``` To exclude the Raspberry Pi from the Kubernetes cluster, set @@ -328,7 +338,7 @@ preferred pod anti-affinity so future Argo CD, Kyverno, Prometheus operator, and kube-state-metrics scheduling does not collapse onto the first worker that joins. PVC-backed monitoring StatefulSets are intentionally treated separately because their retained OpenEBS hostpath volumes are node-local. Run -`./lab.sh move-prometheus-stack-workers` from the Debian host to label existing +`./jeannie move-prometheus-stack-workers` from the Debian host to label existing worker nodes, destroy only the existing `prometheus-stack` Helm release, delete its retained PVC/PV objects, and recreate the stack on the worker selector when you intentionally accept losing that monitoring data. A planned monitoring data @@ -395,7 +405,7 @@ The edge stack uses Traefik as its backend by default and validates subnet-route approval is not automatic in the tailnet policy, the edge deploy will fail clearly instead of silently keeping a broken public path. -For `./lab.sh nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of +For `./jeannie nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of remote SSH targets when more worker nodes exist. Set it to an empty string for a single-node rebuild. @@ -423,13 +433,13 @@ and audits whether the image has both: The private Cosign key, generated password, and downloaded Cosign binary live in `${XDG_DATA_HOME:-$HOME/.local/share}/homelab/` by default so the signing identity survives temporary checkout cleanup. Repo-local image state files still -live under `.lab/`, which is ignored by Git. `./lab.sh apps` creates or reuses +live under `.lab/`, which is ignored by Git. `./jeannie apps` creates or reuses the persistent Cosign key, publishes the public key into the Kyverno namespace as `homelab-cosign-public-key`, builds images with BuildKit SBOM attestations, signs the pushed images, attaches a signed SPDX SBOM predicate, and verifies both artifacts before refreshing the workload Applications. The policy starts in `Audit` mode so an existing live deployment can recover while the current images -are signed; after `./lab.sh apps` verifies both images, change `validationActions` in +are signed; after `./jeannie apps` verifies both images, change `validationActions` in `apps/supply-chain-policy/local-registry-image-policy.yaml` from `Audit` to `Deny` to make it admission-enforcing. Set `COSIGN_PASSWORD` if you want to manage the key password externally; otherwise the Debian runner creates @@ -513,7 +523,7 @@ certificate warning because the trusted certificate is issued for the hostname. The edge stack uses HTTP-01 validation and requests one certificate covering `server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must -be open before `./lab.sh up` runs. Set `TF_VAR_letsencrypt_email` to receive +be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive expiry notices, or leave it empty to register without an email. Set `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. @@ -523,7 +533,7 @@ Add Kubernetes manifests under `apps/` and register them in `bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and self-healing for the app. -The `heimdall` app is intentionally waited on at the end of `./lab.sh apps`. +The `heimdall` app is intentionally waited on at the end of `./jeannie apps`. It runs the LinuxServer.io Heimdall dashboard, persists `/config` on OpenEBS, and seeds tiles for the website, demo apps, Gitea, Grafana, Prometheus, Alertmanager, Argo CD, the local registry, and Heimdall itself. @@ -598,12 +608,12 @@ restart `argocd-repo-server`, and hard-refresh the affected Application. Deploy or refresh the external Gitea container from the Debian host with: ```bash -./lab.sh deploy-gitea +./jeannie deploy-gitea ``` ## Gitea Backups -`./lab.sh up` installs a Debian-host systemd timer named +`./jeannie up` installs a Debian-host systemd timer named `homelab-gitea-backup.timer`. The timer runs daily, SSHes to the configured Gitea host, executes `gitea dump` inside the Gitea Docker container, copies the dump back to Debian, and stores it under `/home/jv/backups/gitea`. The default @@ -618,13 +628,13 @@ does not write into the live Gitea data directory. Run a manual backup from the Debian server with: ```bash -./lab.sh backup-gitea +./jeannie backup-gitea ``` Run the restore drill manually with: ```bash -./lab.sh drill-gitea-restore +./jeannie drill-gitea-restore ``` Useful checks: @@ -646,16 +656,16 @@ a repository-scoped Debian host runner with the label `homelab-debian`. The workflow only blocks automatic deploy for external Gitea service changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`, `install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore` -functions in `lab.sh`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by -default: Actions runs `./lab.sh doctor-versions` when the Debian runner already -has a cluster kubeconfig. If node versions are aligned it runs `./lab.sh apps`; -if kubelet minor drift is detected it runs `./lab.sh rebuild-cluster`; if no -kubeconfig exists it also runs `./lab.sh rebuild-cluster`. +functions in `jeannie`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by +default: Actions runs `./jeannie doctor-versions` when the Debian runner already +has a cluster kubeconfig. If node versions are aligned it runs `./jeannie apps`; +if kubelet minor drift is detected it runs `./jeannie rebuild-cluster`; if no +kubeconfig exists it also runs `./jeannie rebuild-cluster`. Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster` on the runner to force one path. -`./lab.sh bootstrap-gitea-repo` also registers the Debian host SSH public key +`./jeannie bootstrap-gitea-repo` also registers the Debian host SSH public key with the Gitea repository and switches the Debian working copy's `gitea` remote to `ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git`. The default key is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a @@ -677,7 +687,7 @@ Register and start the Debian runner from the Debian server: ```bash cd ~/my-homelab-configs -GITEA_RUNNER_REGISTRATION_TOKEN='' ./lab.sh install-gitea-runner +GITEA_RUNNER_REGISTRATION_TOKEN='' ./jeannie install-gitea-runner ``` The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and @@ -700,12 +710,12 @@ journalctl -u homelab-gitea-runner.service -n 100 --no-pager providers, Helm chart versions, and the pinned tools used by the Gitea Actions workflow. Renovate should open reviewable update branches or PRs only; it must not auto-merge infrastructure changes. Keep app-only dependency updates on the -normal Gitea Actions path, and run `./lab.sh up` manually on the Debian server +normal Gitea Actions path, and run `./jeannie up` manually on the Debian server for platform or provisioning updates. ## Destructive Rebuilds -`./lab.sh nuke` resets kubeadm, containerd runtime state, CNI files, Calico +`./jeannie nuke` resets kubeadm, containerd runtime state, CNI files, Calico links, iptables rules, and local OpenTofu state. It does not delete retained data under `/data/openebs/local`. @@ -735,14 +745,14 @@ Create or refresh the Ollama model on the Debian server before deploying a website image that points at it: ```bash -./lab.sh website-translation-model +./jeannie website-translation-model ``` Ollama must also listen on the Debian host LAN address so Kubernetes pods on other nodes can reach `OLLAMA_HOST=http://192.168.100.73:11434`: ```bash -./lab.sh website-ollama-listen +./jeannie website-ollama-listen ``` The CV page has two client-side presentation modes: @@ -757,22 +767,22 @@ served from a separate `demos-static` artifact under `apps/demos-static` and are published through the `demos-static` Argo CD application. Public traffic reaches them through the edge path at `/demo-apps/`. -`./lab.sh up` builds and pushes two independent images: +`./jeannie up` builds and pushes two independent images: -- a content-hash `php-website` tag generated by `lab.sh` and passed to Argo CD +- a content-hash `php-website` tag generated by `jeannie` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` - Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. -During bootstrap, `lab.sh` hashes `apps/website`, builds +During bootstrap, `jeannie` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains immutable. The Kyverno supply-chain policy mutates admitted pods to the verified image digest, so the workload runs the same digest that was signed and attested. -After `./lab.sh apps`, the live deployment image should be a content-hash tag, +After `./jeannie apps`, the live deployment image should be a content-hash tag, for example `192.168.100.73:30500/php-website:src-...`. If it still shows `php-website:latest`, Argo CD has not rendered the current Application source. Check the `website-production` Application source, sync status, and repository diff --git a/jeannie b/jeannie index 388dbe6..35ac8ca 100755 --- a/jeannie +++ b/jeannie @@ -1925,7 +1925,7 @@ if not isinstance(document, dict): "documentNamespace": f"https://homelab.local/spdx/{safe_ref}/{source_hash}", "creationInfo": { "created": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), - "creators": ["Tool: lab.sh"], + "creators": ["Tool: jeannie"], }, "packages": [ { @@ -3430,6 +3430,180 @@ start_cluster() { echo "Kubernetes runtime start requested. Use 'kubectl get nodes -o wide' to watch readiness." } +status_section() { + printf '\n== %s ==\n' "$1" +} + +status_run() { + local description="$1" + shift + + printf '\n-- %s\n' "${description}" + if ! "$@"; then + printf 'status check failed: %s\n' "${description}" >&2 + fi +} + +status_http() { + local name="$1" + local url="$2" + + if command -v curl >/dev/null 2>&1; then + printf '%-28s ' "${name}" + curl -k -sS -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code} %{url_effective}\n' "${url}" || + printf 'unreachable %s\n' "${url}" + else + printf '%-28s curl not installed\n' "${name}" + fi +} + +status_systemd_units() { + local unit + + for unit in "$@"; do + if systemctl list-unit-files --no-legend "${unit}.service" 2>/dev/null | grep -q .; then + printf '%-28s %s\n' "${unit}" "$(systemctl is-active "${unit}" 2>/dev/null || true)" + else + printf '%-28s not-installed\n' "${unit}" + fi + done +} + +status_compose_stack() { + local name="$1" + local directory="$2" + + printf '\n-- Docker Compose: %s (%s)\n' "${name}" "${directory}" + if [[ ! -d "${directory}" ]]; then + printf 'missing directory\n' + return 0 + fi + if ! command -v docker >/dev/null 2>&1; then + printf 'docker not installed\n' + return 0 + fi + if ! sudo docker compose version >/dev/null 2>&1; then + printf 'docker compose unavailable\n' + return 0 + fi + + (cd "${directory}" && sudo docker compose ps) || true +} + +status_kubernetes() { + status_section "Kubernetes" + + if ! command -v kubectl >/dev/null 2>&1; then + printf 'kubectl not installed\n' + return 0 + fi + if [[ ! -s "${KUBECONFIG_PATH}" ]]; then + printf 'kubeconfig missing: %s\n' "${KUBECONFIG_PATH}" + return 0 + fi + if ! kubectl --kubeconfig "${KUBECONFIG_PATH}" get --raw=/readyz >/dev/null 2>&1; then + printf 'API server not reachable through %s\n' "${KUBECONFIG_PATH}" + return 0 + fi + + status_run "nodes" kubectl --kubeconfig "${KUBECONFIG_PATH}" get nodes -o wide + status_kubernetes_problem_pods + status_run "traefik services" kubectl --kubeconfig "${KUBECONFIG_PATH}" -n traefik get svc,pods -o wide + status_run "website pods" kubectl --kubeconfig "${KUBECONFIG_PATH}" -n website-production get pods -o wide +} + +status_kubernetes_problem_pods() { + local rows + + printf '\n-- pods not Running/Completed\n' + rows="$( + kubectl --kubeconfig "${KUBECONFIG_PATH}" get pods -A --no-headers -o wide | + awk '$4 != "Running" && $4 != "Completed" { print }' + )" + if [[ -n "${rows}" ]]; then + printf '%s\n' "${rows}" + else + printf 'none\n' + fi +} + +status_pimox_workers() { + local pimox_host="${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}" + local pimox_user="${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}" + local pimox_key="${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-/home/jv/.ssh/id_ed25519}}" + local qm_bin="${LAB_PIMOX_QM_BIN:-${TF_VAR_pimox_qm_bin:-/usr/sbin/qm}}" + local worker_count="${LAB_PIMOX_WORKER_COUNT:-$(pimox_worker_count_default)}" + local worker_base_vmid="${LAB_PIMOX_WORKER_BASE_VMID:-9010}" + local worker_skip_indexes="${LAB_PIMOX_SKIP_WORKER_INDEXES:-}" + local index + local vmid + + status_section "Pimox Workers" + + if ! [[ "${worker_count}" =~ ^[0-9]+$ ]]; then + printf 'invalid LAB_PIMOX_WORKER_COUNT: %s\n' "${worker_count}" + return 0 + fi + if ((worker_count == 0)); then + printf 'worker count is 0\n' + return 0 + fi + + for ((index = 1; index <= worker_count; index++)); do + if worker_index_is_skipped "${index}" "${worker_skip_indexes}"; then + printf 'worker index %s skipped\n' "${index}" + continue + fi + vmid=$((worker_base_vmid + index - 1)) + printf 'VM %-6s ' "${vmid}" + pimox_ssh "${pimox_host}" "${pimox_user}" "${pimox_key}" "sudo '${qm_bin}' status '${vmid}'" 2>/dev/null || + printf 'unreachable or missing\n' + done +} + +status_report() { + require_debian_server "status" + + status_section "Host" + printf 'hostname: %s\n' "$(hostname -f 2>/dev/null || hostname)" + printf 'date: %s\n' "$(date -Is)" + printf 'uptime: %s\n' "$(uptime -p 2>/dev/null || uptime)" + status_run "memory" free -h + status_run "disk" df -h / /data /data/docker /data/openebs/local + + status_section "Systemd" + status_systemd_units ssh docker containerd kubelet tailscaled homelab-gitea-runner + + status_section "Docker" + if command -v docker >/dev/null 2>&1; then + status_run "docker containers" sudo docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}' + else + printf 'docker not installed\n' + fi + status_compose_stack "gitea" "/data/homelab-gitea" + status_compose_stack "arr-stack" "${REPO_ROOT}/infra/arr-stack" + + status_kubernetes + status_pimox_workers + + status_section "Tailscale" + if command -v tailscale >/dev/null 2>&1; then + status_run "tailscale ip" tailscale ip -4 + status_run "tailscale peers" tailscale status + else + printf 'tailscale not installed\n' + fi + + status_section "HTTP" + status_http "website public" "https://lab2025.duckdns.org/" + status_http "gitea public" "https://lab2025.duckdns.org/git/" + status_http "gitea local" "http://127.0.0.1:3000/" + status_http "traefik lb" "http://192.168.100.240/" + status_http "arr radarr" "http://127.0.0.1:7878/" + status_http "arr sonarr" "http://127.0.0.1:8989/" + status_http "arr prowlarr" "http://127.0.0.1:9696/" +} + nuke() { local worker_ssh_targets local worker_targets @@ -3584,6 +3758,9 @@ case "${1:-}" in start-cluster) start_cluster ;; + status) + status_report + ;; apps) apps ;; @@ -3621,7 +3798,7 @@ case "${1:-}" in nuke ;; *) - echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|apps|website-translation-model|website-ollama-listen|deploy-gitea|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|openwrt|nuke}" + echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|deploy-gitea|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|openwrt|nuke}" exit 1 ;; esac