Fix scorecard warning summaries

.gitea/workflows/homelab-main.yml

@@ -4,10 +4,35 @@ name: Homelab Main
   push:
     branches:
       - main
+      - dev
 
 jobs:
+  validate:
+    runs-on: homelab-debian
+
+    steps:
+      - name: Check out repository
+        run: |
+          set -euo pipefail
+
+          find . -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
+          git init
+          git remote add origin ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git
+          git fetch --prune origin +refs/heads/main:refs/remotes/origin/main +refs/heads/dev:refs/remotes/origin/dev +refs/tags/*:refs/tags/*
+          git checkout --force "${{ gitea.sha }}"
+          git config --global --add safe.directory "$PWD"
+
+      - name: Validate repository
+        run: |
+          set -euo pipefail
+
+          ./jeannie validate
+
   deploy:
     runs-on: homelab-debian
+    needs:
+      - validate
+    if: ${{ gitea.ref == 'refs/heads/main' }}
 
     steps:
       - name: Check out repository
@@ -21,12 +46,6 @@ jobs:
           git checkout --force "${{ gitea.sha }}"
           git config --global --add safe.directory "$PWD"
 
-      - name: Check generated docs
-        run: |
-          set -euo pipefail
-
-          scripts/render-docs --check
-
       - name: Block automatic deploy for external Gitea changes
         run: |
           set -euo pipefail

README.md

@@ -269,12 +269,153 @@ hostname.
 Run a read-only health snapshot from the Debian server with:
 
 ```bash
+./jeannie help
 ./jeannie status
+./jeannie capacity
+./jeannie recover-plan
+./jeannie recover-power --dry-run
+./jeannie scorecard
+./jeannie gitops-status
+./jeannie cert-check
+./jeannie grafana-dashboards list
+./jeannie release-snapshot
+./jeannie backup-status
+./jeannie synthetic-checks
+./jeannie resource-budget
+./jeannie control-plane status
+./jeannie artifact-cache status
+./jeannie blockchain-devnet status
+./jeannie blockchain-test
+./jeannie blockchain-wallet instructions
+./jeannie golden-ledger check
+./jeannie route-inventory
+./jeannie workers list
+./jeannie change-journal list
+./jeannie map
 ```
 
+`help` prints the grouped command reference in pipeline order, from inventory
+and bootstrap through cluster lifecycle, reports, observability, recovery,
+access, apps, security, AI, and destructive commands.
+
 It reports host memory/disk, systemd services, Docker Compose stacks,
 Kubernetes health when the API is reachable, Pimox worker VM status, RPi
-services, Tailscale, and key local/public HTTP endpoints.
+services, Tailscale, key local/public HTTP endpoints, and "what broke" signals
+such as deployment readiness, pod restarts, node pressure, disk pressure,
+Traefik 5xx/404 log evidence, and recent Gitea errors.
+
+High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node
+readiness, pod restart storms, unavailable deployments, node/PVC storage
+pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape
+coverage. Keep this set small so alerts remain actionable.
+
+Application egress is controlled with Kubernetes NetworkPolicies where the lab
+owns the workload. The website namespace defaults to denied egress and allows
+only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is
+isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS
+for controlled practice targets.
+
+`capacity` is the placement report for the hardware you already have. By
+default it uses the compact report renderer and shows summarized Debian,
+Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use
+`./jeannie capacity --details`, `--json`, `--only problems`, or
+`--verbose` for the older full capacity dump.
+
+`recover-plan` prints the disaster recovery order and lightweight prerequisite
+checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and
+edge verification.
+
+`recover-power` runs the post-outage recovery sequence in dependency order:
+Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and
+edge/public checks. Use `recover-power --dry-run` to preview the sequence.
+
+`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster
+health, edge health, capacity pressure, security posture, public certificates,
+inventory, and docs freshness into a compact pass/warn/fail report.
+
+Report-oriented commands can share the reusable renderer in
+`scripts/report-render`. It reads status TSV rows and prints compact grouped
+summaries by default, with details or JSON available for future TUI/web output.
+Rows may include separate `fix`, `graph`, and `query` fields so a warning can
+point to both the next `jeannie` command and the matching Grafana panel or
+Explore query.
+
+`gitops-status` prints a focused Argo CD view: application sync and health,
+out-of-sync or degraded apps, repository secret presence, recent Argo CD events,
+and recent repo-server/application-controller errors.
+
+`grafana-dashboards` manages repo-provisioned Grafana dashboards under
+`bootstrap/platform/grafana-dashboards`. `./jeannie up` provisions
+them through the platform stack; `./jeannie grafana-dashboards apply`
+rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional
+blockchain devnet dashboard uses metrics from
+`infra/blockchain-devnet/metrics/exporter.py`.
+
+`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea
+HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory.
+
+`release-snapshot` writes a timestamped pre-change report under the homelab
+state directory with Git state, Kubernetes/Argo CD/Helm state, workload images,
+public URL status, and pointers to the latest Gitea/OpenTofu backups.
+
+`backup-status` reports freshness for Gitea backups, OpenTofu state backups,
+restore drill reports, and repo-managed Pi-hole restore inputs. The main
+`status` cascade includes this as a warning signal.
+
+`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API,
+Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The
+registry push/pull smoke test is opt-in with
+`LAB_SYNTHETIC_REGISTRY_PUSH=true`.
+
+`resource-budget` checks the report-only policy in `infra/resource-budgets.yml`
+against Kubernetes pod requests/limits and Debian host disk/memory signals. Use
+it before adding heavier workloads so the HP laptop remains usable. By default
+it summarizes missing requests/limits by namespace; use
+`./jeannie resource-budget --details` for every pod/container.
+
+`control-plane` shows or manages the Debian Kubernetes control-plane scheduling
+taint. After Pimox workers are stable, use `./jeannie control-plane taint`
+to keep normal app/data-plane pods off the Debian host while control-plane and
+DaemonSet workloads continue using their normal tolerations.
+
+`artifact-cache` manages an optional Debian-host cache stack in
+`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through
+cache. Use `artifact-cache instructions` for client configuration snippets.
+
+`blockchain-devnet` manages an optional local Ethereum Anvil devnet under
+`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability
+learning only; never use real seed phrases or mainnet keys.
+
+`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts
+with a simple `Counter` contract and ownership tests so there is a known-good
+baseline before practicing vulnerable contracts.
+
+`blockchain-wallet` provides disposable dev-wallet generation, address
+derivation, and message-signing practice. It is documented in
+`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys.
+
+`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the
+reviewable record of template VMID, storage, OS release, Kubernetes pins,
+runtime versions, and build metadata for Pimox worker golden images.
+
+`route-inventory` reports Kubernetes ingress hosts, paths, backend services,
+TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and
+missing monitors are easier to spot.
+
+`workers` provides named Kubernetes/Pimox worker lifecycle operations:
+`list`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and `rebalance`.
+Use `recreate-plan <index>` before replacing one broken Pimox worker so the
+cluster drain, VM stop, and `./jeannie up` path stay ordered.
+
+`change-journal` lists local journal entries written before risky commands such
+as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`,
+`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live
+under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch,
+dirty file count, and command details.
+
+`map` prints a dependency map from the canonical inventory, covering public
+entry, GitOps automation, cluster foundation, DNS, observability, and security
+learning paths. `./jeannie map --dot` emits Graphviz DOT.
 
 Focused doctor commands run narrower read-only checks and print the most likely
 next step:
@@ -743,11 +884,18 @@ Run the restore drill manually with:
 
 ```bash
 ./jeannie drill-gitea-restore
+./jeannie drill-restore
+./jeannie drill-pihole-restore
 ```
 
+`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
+config inputs, the latest OpenTofu state backup archive when present, and the
+GitOps rebuild path. It does not replace running services or mutate Kubernetes.
+
 Useful checks:
 
 ```bash
+./jeannie backup-status
 systemctl list-timers homelab-gitea-backup.timer
 systemctl list-timers homelab-gitea-restore-drill.timer
 sudo systemctl start homelab-gitea-backup.service
@@ -758,8 +906,48 @@ ls -lh /home/jv/backups/gitea-restore-drills
 ## Gitea Actions
 
 This repo includes a Gitea Actions workflow at
-`.gitea/workflows/homelab-main.yml`. It runs only on pushes to `main` and targets
-a repository-scoped Debian host runner with the label `homelab-debian`.
+`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
+`main`, and deploys only from `main`. That keeps the promotion path simple:
+
+```text
+dev -> ./jeannie validate in Gitea Actions -> main -> Argo CD sync
+```
+
+The workflow targets a repository-scoped Debian host runner with the label
+`homelab-debian`.
+
+Run the same validation locally before promoting:
+
+```bash
+./jeannie validate
+```
+
+## Identity and access audit
+
+Run a read-only access inventory from the Debian server with:
+
+```bash
+./jeannie access-audit
+```
+
+The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
+Tailscale ACL policy validation, current Kubernetes authorization, broad
+cluster-admin bindings, automation service accounts, and Gitea runner status.
+It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
+separate identities:
+
+- admin kubeconfig only on the Debian control host
+- read-only kubeconfig for dashboards and audits
+- separate service accounts for automation
+- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
+- repo-managed Tailscale ACLs
+
+The read-only Kubernetes identity is managed in `apps/access-control`. After
+that app syncs, generate a separate read-only kubeconfig on the Debian host:
+
+```bash
+./jeannie kubeconfig-readonly
+```
 
 The workflow only blocks automatic deploy for external Gitea service
 changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,

README.md.tmpl

@@ -269,12 +269,153 @@ hostname.
 Run a read-only health snapshot from the Debian server with:
 
 ```bash
+./{{ main_script }} help
 ./{{ main_script }} status
+./{{ main_script }} capacity
+./{{ main_script }} recover-plan
+./{{ main_script }} recover-power --dry-run
+./{{ main_script }} scorecard
+./{{ main_script }} gitops-status
+./{{ main_script }} cert-check
+./{{ main_script }} grafana-dashboards list
+./{{ main_script }} release-snapshot
+./{{ main_script }} backup-status
+./{{ main_script }} synthetic-checks
+./{{ main_script }} resource-budget
+./{{ main_script }} control-plane status
+./{{ main_script }} artifact-cache status
+./{{ main_script }} blockchain-devnet status
+./{{ main_script }} blockchain-test
+./{{ main_script }} blockchain-wallet instructions
+./{{ main_script }} golden-ledger check
+./{{ main_script }} route-inventory
+./{{ main_script }} workers list
+./{{ main_script }} change-journal list
+./{{ main_script }} map
 ```
 
+`help` prints the grouped command reference in pipeline order, from inventory
+and bootstrap through cluster lifecycle, reports, observability, recovery,
+access, apps, security, AI, and destructive commands.
+
 It reports host memory/disk, systemd services, Docker Compose stacks,
 Kubernetes health when the API is reachable, Pimox worker VM status, RPi
-services, Tailscale, and key local/public HTTP endpoints.
+services, Tailscale, key local/public HTTP endpoints, and "what broke" signals
+such as deployment readiness, pod restarts, node pressure, disk pressure,
+Traefik 5xx/404 log evidence, and recent Gitea errors.
+
+High-signal Prometheus alerts live in `apps/homelab-alerts`. They cover node
+readiness, pod restart storms, unavailable deployments, node/PVC storage
+pressure, Traefik 5xx spikes, website availability, and Gitea/Pi-hole scrape
+coverage. Keep this set small so alerts remain actionable.
+
+Application egress is controlled with Kubernetes NetworkPolicies where the lab
+owns the workload. The website namespace defaults to denied egress and allows
+only DNS, Redis, and the Debian Ollama endpoint. The security lab namespace is
+isolated from the cluster and LAN while allowing DNS plus outbound HTTP/HTTPS
+for controlled practice targets.
+
+`capacity` is the placement report for the hardware you already have. By
+default it uses the compact report renderer and shows summarized Debian,
+Kubernetes, Pimox, RPi, and placement signals without dumping full details. Use
+`./{{ main_script }} capacity --details`, `--json`, `--only problems`, or
+`--verbose` for the older full capacity dump.
+
+`recover-plan` prints the disaster recovery order and lightweight prerequisite
+checks, from Debian and Gitea through DNS, Pimox, Kubernetes, GitOps apps, and
+edge verification.
+
+`recover-power` runs the post-outage recovery sequence in dependency order:
+Debian runtime, Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and
+edge/public checks. Use `recover-power --dry-run` to preview the sequence.
+
+`scorecard` rolls up backup readiness, GitOps health, DNS/RPi health, cluster
+health, edge health, capacity pressure, security posture, public certificates,
+inventory, and docs freshness into a compact pass/warn/fail report.
+
+Report-oriented commands can share the reusable renderer in
+`scripts/report-render`. It reads status TSV rows and prints compact grouped
+summaries by default, with details or JSON available for future TUI/web output.
+Rows may include separate `fix`, `graph`, and `query` fields so a warning can
+point to both the next `jeannie` command and the matching Grafana panel or
+Explore query.
+
+`gitops-status` prints a focused Argo CD view: application sync and health,
+out-of-sync or degraded apps, repository secret presence, recent Argo CD events,
+and recent repo-server/application-controller errors.
+
+`grafana-dashboards` manages repo-provisioned Grafana dashboards under
+`bootstrap/platform/grafana-dashboards`. `./{{ main_script }} up` provisions
+them through the platform stack; `./{{ main_script }} grafana-dashboards apply`
+rebuilds just the dashboard ConfigMaps and restarts Grafana. The optional
+blockchain devnet dashboard uses metrics from
+`infra/blockchain-devnet/metrics/exporter.py`.
+
+`cert-check` verifies public DNS, TLS certificate expiry, public website/Gitea
+HTTP status, and DuckDNS-to-OCI edge IP drift from the canonical inventory.
+
+`release-snapshot` writes a timestamped pre-change report under the homelab
+state directory with Git state, Kubernetes/Argo CD/Helm state, workload images,
+public URL status, and pointers to the latest Gitea/OpenTofu backups.
+
+`backup-status` reports freshness for Gitea backups, OpenTofu state backups,
+restore drill reports, and repo-managed Pi-hole restore inputs. The main
+`status` cascade includes this as a warning signal.
+
+`synthetic-checks` runs end-to-end probes for Gitea HTTP/SSH, registry API,
+Pi-hole DNS, Traefik LAN HTTP, and temporary Kubernetes DNS/ingress pods. The
+registry push/pull smoke test is opt-in with
+`LAB_SYNTHETIC_REGISTRY_PUSH=true`.
+
+`resource-budget` checks the report-only policy in `infra/resource-budgets.yml`
+against Kubernetes pod requests/limits and Debian host disk/memory signals. Use
+it before adding heavier workloads so the HP laptop remains usable. By default
+it summarizes missing requests/limits by namespace; use
+`./{{ main_script }} resource-budget --details` for every pod/container.
+
+`control-plane` shows or manages the Debian Kubernetes control-plane scheduling
+taint. After Pimox workers are stable, use `./{{ main_script }} control-plane taint`
+to keep normal app/data-plane pods off the Debian host while control-plane and
+DaemonSet workloads continue using their normal tolerations.
+
+`artifact-cache` manages an optional Debian-host cache stack in
+`infra/artifact-cache` with apt-cacher-ng and a Docker Hub registry pull-through
+cache. Use `artifact-cache instructions` for client configuration snippets.
+
+`blockchain-devnet` manages an optional local Ethereum Anvil devnet under
+`infra/blockchain-devnet`. It is for contract, wallet, RPC, and observability
+learning only; never use real seed phrases or mainnet keys.
+
+`blockchain-test` runs the Foundry tests in `labs/blockchain`. The lab starts
+with a simple `Counter` contract and ownership tests so there is a known-good
+baseline before practicing vulnerable contracts.
+
+`blockchain-wallet` provides disposable dev-wallet generation, address
+derivation, and message-signing practice. It is documented in
+`labs/blockchain/WALLET_LAB.md`; never use real seed phrases or funded keys.
+
+`golden-ledger` shows or validates `infra/pimox/golden-image-ledger.yml`, the
+reviewable record of template VMID, storage, OS release, Kubernetes pins,
+runtime versions, and build metadata for Pimox worker golden images.
+
+`route-inventory` reports Kubernetes ingress hosts, paths, backend services,
+TLS coverage, and visible Uptime Kuma monitor coverage so stale routes and
+missing monitors are easier to spot.
+
+`workers` provides named Kubernetes/Pimox worker lifecycle operations:
+`list`, `start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and `rebalance`.
+Use `recreate-plan <index>` before replacing one broken Pimox worker so the
+cluster drain, VM stop, and `./{{ main_script }} up` path stay ordered.
+
+`change-journal` lists local journal entries written before risky commands such
+as `up`, `apps`, `deploy-gitea`, `rpi-services`, `stop-cluster`,
+`start-cluster`, `rebuild-cluster`, `release-snapshot`, and `nuke`. Entries live
+under `${HOMELAB_STATE_DIR}/change-journal` and record Git revision, branch,
+dirty file count, and command details.
+
+`map` prints a dependency map from the canonical inventory, covering public
+entry, GitOps automation, cluster foundation, DNS, observability, and security
+learning paths. `./{{ main_script }} map --dot` emits Graphviz DOT.
 
 Focused doctor commands run narrower read-only checks and print the most likely
 next step:
@@ -743,11 +884,18 @@ Run the restore drill manually with:
 
 ```bash
 ./{{ main_script }} drill-gitea-restore
+./{{ main_script }} drill-restore
+./{{ main_script }} drill-pihole-restore
 ```
 
+`drill-restore` validates the latest Gitea backup archive, Pi-hole repo-managed
+config inputs, the latest OpenTofu state backup archive when present, and the
+GitOps rebuild path. It does not replace running services or mutate Kubernetes.
+
 Useful checks:
 
 ```bash
+./{{ main_script }} backup-status
 systemctl list-timers homelab-gitea-backup.timer
 systemctl list-timers homelab-gitea-restore-drill.timer
 sudo systemctl start homelab-gitea-backup.service
@@ -758,8 +906,48 @@ ls -lh /home/jv/backups/gitea-restore-drills
 ## Gitea Actions
 
 This repo includes a Gitea Actions workflow at
-`.gitea/workflows/homelab-main.yml`. It runs only on pushes to `main` and targets
-a repository-scoped Debian host runner with the label `homelab-debian`.
+`.gitea/workflows/homelab-main.yml`. It runs validation on pushes to `dev` and
+`main`, and deploys only from `main`. That keeps the promotion path simple:
+
+```text
+dev -> ./{{ main_script }} validate in Gitea Actions -> main -> Argo CD sync
+```
+
+The workflow targets a repository-scoped Debian host runner with the label
+`homelab-debian`.
+
+Run the same validation locally before promoting:
+
+```bash
+./{{ main_script }} validate
+```
+
+## Identity and access audit
+
+Run a read-only access inventory from the Debian server with:
+
+```bash
+./{{ main_script }} access-audit
+```
+
+The audit reports configured SSH targets and key file modes, Git/Gitea remotes,
+Tailscale ACL policy validation, current Kubernetes authorization, broad
+cluster-admin bindings, automation service accounts, and Gitea runner status.
+It does not create users, tokens, kubeconfigs, or keys. Use it to move toward
+separate identities:
+
+- admin kubeconfig only on the Debian control host
+- read-only kubeconfig for dashboards and audits
+- separate service accounts for automation
+- explicit SSH inventory for Debian, RPi, Pimox, and OCI edge
+- repo-managed Tailscale ACLs
+
+The read-only Kubernetes identity is managed in `apps/access-control`. After
+that app syncs, generate a separate read-only kubeconfig on the Debian host:
+
+```bash
+./{{ main_script }} kubeconfig-readonly
+```
 
 The workflow only blocks automatic deploy for external Gitea service
 changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,

apps/access-control/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - readonly-rbac.yaml

apps/access-control/readonly-rbac.yaml

@@ -0,0 +1,108 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: homelab-access
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: homelab-readonly
+  namespace: homelab-access
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: homelab-readonly
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - componentstatuses
+      - configmaps
+      - endpoints
+      - events
+      - namespaces
+      - nodes
+      - persistentvolumeclaims
+      - persistentvolumes
+      - pods
+      - pods/log
+      - replicationcontrollers
+      - resourcequotas
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - apps
+    resources:
+      - daemonsets
+      - deployments
+      - replicasets
+      - statefulsets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - batch
+    resources:
+      - cronjobs
+      - jobs
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - networking.k8s.io
+    resources:
+      - ingresses
+      - networkpolicies
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - monitoring.coreos.com
+    resources:
+      - prometheusrules
+      - servicemonitors
+      - podmonitors
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - argoproj.io
+    resources:
+      - applications
+      - appprojects
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - kyverno.io
+      - policies.kyverno.io
+    resources:
+      - clusterpolicies
+      - policies
+      - imagevalidatingpolicies
+    verbs:
+      - get
+      - list
+      - watch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: homelab-readonly
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: homelab-readonly
+subjects:
+  - kind: ServiceAccount
+    name: homelab-readonly
+    namespace: homelab-access

apps/homelab-alerts/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - prometheus-rules.yaml

apps/homelab-alerts/prometheus-rules.yaml

@@ -0,0 +1,90 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: homelab-high-signal-alerts
+  namespace: monitoring
+  labels:
+    release: prometheus-stack
+    role: alert-rules
+spec:
+  groups:
+    - name: homelab.availability
+      rules:
+        - alert: HomelabKubernetesNodeNotReady
+          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Kubernetes node {{ $labels.node }} is not Ready"
+            description: "A node has been NotReady for 10 minutes. Run ./jeannie doctor-cluster."
+        - alert: HomelabPodRestartStorm
+          expr: increase(kube_pod_container_status_restarts_total[15m]) >= 3
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Pod restart storm in {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "A container restarted at least 3 times in 15 minutes. Check pod logs and recent deployments."
+        - alert: HomelabDeploymentUnavailable
+          expr: kube_deployment_status_replicas_available < kube_deployment_spec_replicas
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} is unavailable"
+            description: "Available replicas are below desired replicas for 10 minutes."
+    - name: homelab.storage
+      rules:
+        - alert: HomelabNodeDiskPressure
+          expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Node {{ $labels.node }} reports DiskPressure"
+            description: "DiskPressure is active. Run ./jeannie capacity and inspect image/PVC/log usage."
+        - alert: HomelabPersistentVolumeAlmostFull
+          expr: (1 - kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 85
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is over 85% full"
+            description: "PVC capacity is near exhaustion. Review app storage before it becomes an outage."
+    - name: homelab.edge
+      rules:
+        - alert: HomelabTraefik5xxSpike
+          expr: sum(rate(traefik_service_requests_total{code=~"5.."}[5m])) > 0.05
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Traefik is returning elevated 5xx responses"
+            description: "Public edge or backend service may be unhealthy. Run ./jeannie doctor-edge."
+        - alert: HomelabWebsiteDown
+          expr: absent(kube_deployment_status_replicas_available{namespace="website-production",deployment="php-website-deployment"} > 0)
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Website deployment has no available replicas"
+            description: "The public website has no available Kubernetes replicas."
+        - alert: HomelabGiteaRouteDown
+          expr: absent(up{job=~".*gitea.*"} == 1)
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No Gitea scrape target is healthy"
+            description: "If Gitea is intentionally not scraped, keep Uptime Kuma as the source of truth; otherwise check ./jeannie doctor-gitea."
+    - name: homelab.dns
+      rules:
+        - alert: HomelabPiHoleDNSDown
+          expr: absent(up{job=~".*pihole.*"} == 1)
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No Pi-hole scrape target is healthy"
+            description: "If Pi-hole is intentionally not scraped, Uptime Kuma still monitors DNS. Run ./jeannie doctor-rpi."

apps/security-lab/README.md

@@ -0,0 +1,16 @@
+# Security Lab
+
+This namespace contains intentionally vulnerable applications for defensive
+testing practice. It is not exposed through Traefik or the public edge.
+
+Use port-forwarding from the Debian host:
+
+```sh
+kubectl -n security-lab port-forward svc/juice-shop 3000:3000
+kubectl -n security-lab port-forward svc/webgoat 8080:8080
+```
+
+Then point ZAP, nuclei, or a browser at `http://127.0.0.1:3000` or
+`http://127.0.0.1:8080`.
+
+Keep this namespace isolated. Do not add public ingress for these workloads.

apps/security-lab/juice-shop.yaml

@@ -0,0 +1,52 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: juice-shop
+  namespace: security-lab
+  labels:
+    app.kubernetes.io/name: juice-shop
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: juice-shop
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: juice-shop
+        homelab.dev/security-lab-app: "true"
+    spec:
+      automountServiceAccountToken: false
+      containers:
+        - name: juice-shop
+          image: bkimminich/juice-shop:latest
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: http
+              containerPort: 3000
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: "1"
+              memory: 768Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: juice-shop
+  namespace: security-lab
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: juice-shop
+  ports:
+    - name: http
+      port: 3000
+      targetPort: http

apps/security-lab/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - network-policy.yaml
+  - juice-shop.yaml
+  - webgoat.yaml

apps/security-lab/namespace.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: security-lab
+  labels:
+    homelab.dev/security-lab: "true"
+    pod-security.kubernetes.io/enforce: baseline
+    pod-security.kubernetes.io/audit: restricted
+    pod-security.kubernetes.io/warn: restricted

apps/security-lab/network-policy.yaml

@@ -0,0 +1,74 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: security-lab-default-deny-egress
+  namespace: security-lab
+spec:
+  podSelector: {}
+  policyTypes:
+    - Egress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: security-lab-allow-web-egress
+  namespace: security-lab
+spec:
+  podSelector:
+    matchLabels:
+      homelab.dev/security-lab-app: "true"
+  policyTypes:
+    - Egress
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+            except:
+              - 10.96.0.0/12
+              - 192.168.100.0/24
+      ports:
+        - protocol: TCP
+          port: 80
+        - protocol: TCP
+          port: 443
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: security-lab-default-deny-ingress
+  namespace: security-lab
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: security-lab-allow-port-forward
+  namespace: security-lab
+spec:
+  podSelector:
+    matchLabels:
+      homelab.dev/security-lab-app: "true"
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: TCP
+          port: 3000
+        - protocol: TCP
+          port: 8080

apps/security-lab/webgoat.yaml

@@ -0,0 +1,52 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: webgoat
+  namespace: security-lab
+  labels:
+    app.kubernetes.io/name: webgoat
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: webgoat
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: webgoat
+        homelab.dev/security-lab-app: "true"
+    spec:
+      automountServiceAccountToken: false
+      containers:
+        - name: webgoat
+          image: webgoat/webgoat:latest
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: http
+              containerPort: 8080
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: webgoat
+  namespace: security-lab
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: webgoat
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

apps/supply-chain-policy/kustomization.yaml

@@ -3,3 +3,4 @@ kind: Kustomization
 resources:
   - kyverno-cosign-key-rbac.yaml
   - local-registry-image-policy.yaml
+  - pod-hardening-audit.yaml

apps/supply-chain-policy/pod-hardening-audit.yaml

@@ -0,0 +1,164 @@
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: homelab-pod-hardening-audit
+  annotations:
+    policies.kyverno.io/title: Homelab Pod Hardening Audit
+    policies.kyverno.io/category: Pod Security
+    policies.kyverno.io/severity: medium
+    policies.kyverno.io/subject: Pod
+    policies.kyverno.io/description: >-
+      Audit pod settings that commonly increase compromise blast radius. These
+      rules intentionally start in Audit mode so enforcement can be promoted one
+      rule at a time after observing current workloads.
+spec:
+  validationFailureAction: Audit
+  background: true
+  rules:
+    - name: disallow-privileged-containers
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+      exclude:
+        any:
+          - resources:
+              namespaces:
+                - kube-system
+                - tigera-operator
+                - calico-system
+                - metallb-system
+                - openebs
+                - kyverno
+      validate:
+        message: "Privileged containers should be avoided unless explicitly justified."
+        pattern:
+          spec:
+            =(ephemeralContainers):
+              - =(securityContext):
+                  =(privileged): "false"
+            =(initContainers):
+              - =(securityContext):
+                  =(privileged): "false"
+            containers:
+              - =(securityContext):
+                  =(privileged): "false"
+    - name: disallow-privilege-escalation
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+      exclude:
+        any:
+          - resources:
+              namespaces:
+                - kube-system
+                - tigera-operator
+                - calico-system
+                - metallb-system
+                - openebs
+                - kyverno
+      validate:
+        message: "Containers should set allowPrivilegeEscalation=false."
+        pattern:
+          spec:
+            =(ephemeralContainers):
+              - securityContext:
+                  allowPrivilegeEscalation: "false"
+            =(initContainers):
+              - securityContext:
+                  allowPrivilegeEscalation: "false"
+            containers:
+              - securityContext:
+                  allowPrivilegeEscalation: "false"
+    - name: disallow-hostpath-volumes
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+      exclude:
+        any:
+          - resources:
+              namespaces:
+                - kube-system
+                - tigera-operator
+                - calico-system
+                - metallb-system
+                - openebs
+                - kyverno
+      validate:
+        message: "hostPath volumes should be avoided outside platform components."
+        pattern:
+          spec:
+            =(volumes):
+              - X(hostPath): "null"
+    - name: require-resource-requests-and-limits
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+      exclude:
+        any:
+          - resources:
+              namespaces:
+                - kube-system
+                - tigera-operator
+                - calico-system
+                - metallb-system
+                - openebs
+                - kyverno
+      validate:
+        message: "Containers should define CPU/memory requests and memory limits."
+        pattern:
+          spec:
+            containers:
+              - resources:
+                  requests:
+                    cpu: "?*"
+                    memory: "?*"
+                  limits:
+                    memory: "?*"
+---
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: homelab-image-tag-audit
+  annotations:
+    policies.kyverno.io/title: Homelab Image Tag Audit
+    policies.kyverno.io/category: Software Supply Chain Security
+    policies.kyverno.io/severity: medium
+    policies.kyverno.io/subject: Pod
+    policies.kyverno.io/description: >-
+      Audit mutable image references. Enforcement should wait until learning
+      workloads and third-party apps have explicit version pins or exceptions.
+spec:
+  validationFailureAction: Audit
+  background: true
+  rules:
+    - name: disallow-latest-tag
+      match:
+        any:
+          - resources:
+              kinds:
+                - Pod
+      validate:
+        message: "Use explicit image versions instead of the mutable latest tag."
+        foreach:
+          - list: "request.object.spec.containers"
+            deny:
+              conditions:
+                any:
+                  - key: "{{ element.image }}"
+                    operator: Equals
+                    value: "*:latest"
+          - list: "request.object.spec.initContainers || `[]`"
+            deny:
+              conditions:
+                any:
+                  - key: "{{ element.image }}"
+                    operator: Equals
+                    value: "*:latest"

apps/website/kustomization.yaml

@@ -2,4 +2,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - namespace.yaml
+  - network-policy.yaml
   - web-app.yaml

apps/website/network-policy.yaml

@@ -0,0 +1,56 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: website-default-deny-egress
+  namespace: website-production
+spec:
+  podSelector: {}
+  policyTypes:
+    - Egress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: website-allow-required-egress
+  namespace: website-production
+spec:
+  podSelector:
+    matchLabels:
+      app: php-website
+  policyTypes:
+    - Egress
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    - to:
+        - podSelector:
+            matchLabels:
+              app: website-redis
+      ports:
+        - protocol: TCP
+          port: 6379
+    - to:
+        - ipBlock:
+            cidr: 192.168.100.73/32
+      ports:
+        - protocol: TCP
+          port: 11434
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: website-redis-deny-egress
+  namespace: website-production
+spec:
+  podSelector:
+    matchLabels:
+      app: website-redis
+  policyTypes:
+    - Egress

bootstrap/apps/variables.tf

@@ -39,6 +39,15 @@ variable "applications" {
       self_heal        = true
       create_namespace = true
     }
+    access-control = {
+      project          = "default"
+      path             = "apps/access-control"
+      namespace        = "homelab-access"
+      target_revision  = "main"
+      prune            = true
+      self_heal        = true
+      create_namespace = true
+    }
     website-production = {
       project          = "default"
       path             = "apps/website"
@@ -84,5 +93,23 @@ variable "applications" {
       self_heal        = true
       create_namespace = false
     }
+    homelab-alerts = {
+      project          = "default"
+      path             = "apps/homelab-alerts"
+      namespace        = "monitoring"
+      target_revision  = "main"
+      prune            = true
+      self_heal        = true
+      create_namespace = false
+    }
+    security-lab = {
+      project          = "default"
+      path             = "apps/security-lab"
+      namespace        = "security-lab"
+      target_revision  = "main"
+      prune            = true
+      self_heal        = true
+      create_namespace = true
+    }
   }
 }

bootstrap/host/README.md

@@ -30,6 +30,7 @@ The playbook:
   the current Debian release
 - installs user-level pipx CLI tools such as `yt-dlp`, plus runtime helpers
   such as `ffmpeg` and `nodejs`
+- installs Lynis for report-only host audits through `./jeannie security-host`
 - installs `age` and `sops` for encrypted repo-managed secrets
 - installs and configures Ollama for homelab-local AI helpers when
   `debian_pc_install_ollama` is true

bootstrap/host/group_vars/debian_pc.yml

@@ -57,6 +57,7 @@ debian_pc_base_packages:
   - git
   - gnupg
   - gpg
+  - lynis
   - nodejs
   - python3
   - python3-apt

bootstrap/platform/grafana-dashboards/homelab-availability.json

@@ -0,0 +1,295 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "sum(kube_node_status_condition{condition=\"Ready\",status=\"true\"})",
+          "legendFormat": "ready nodes",
+          "refId": "A"
+        }
+      ],
+      "title": "Ready Nodes",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 3
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "sum(kube_deployment_spec_replicas - kube_deployment_status_replicas_available)",
+          "legendFormat": "missing replicas",
+          "refId": "A"
+        }
+      ],
+      "title": "Missing Deployment Replicas",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "sum(increase(kube_pod_container_status_restarts_total[15m]))",
+          "legendFormat": "restarts",
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Restarts In 15m",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "sum by (code) (rate(traefik_service_requests_total{code=~\"4..|5..\"}[5m]))",
+          "legendFormat": "{{code}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Traefik Error Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "legendFormat": "{{job}} {{instance}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Scrape Target Availability",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "homelab",
+    "jeannie",
+    "availability"
+  ],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Homelab Availability",
+  "uid": "homelab-availability",
+  "version": 1,
+  "weekStart": ""
+}

bootstrap/platform/grafana-dashboards/homelab-blockchain-devnet.json

@@ -0,0 +1,247 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "text": "down"
+                },
+                "1": {
+                  "text": "up"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "homelab_blockchain_rpc_up",
+          "legendFormat": "rpc",
+          "refId": "A"
+        }
+      ],
+      "title": "RPC Health",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 6,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "homelab_blockchain_latest_block",
+          "legendFormat": "latest block",
+          "refId": "A"
+        }
+      ],
+      "title": "Latest Block",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "homelab_blockchain_gas_price_wei",
+          "legendFormat": "wei",
+          "refId": "A"
+        }
+      ],
+      "title": "Gas Price",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "id": 4,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "expr": "homelab_blockchain_peer_count",
+          "legendFormat": "peers",
+          "refId": "A"
+        }
+      ],
+      "title": "Peer Count",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 8
+      },
+      "id": 5,
+      "targets": [
+        {
+          "expr": "homelab_blockchain_rpc_latency_seconds",
+          "legendFormat": "{{ method }}",
+          "refId": "A"
+        }
+      ],
+      "title": "RPC Latency",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "10s",
+  "schemaVersion": 39,
+  "tags": [
+    "homelab",
+    "blockchain"
+  ],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Homelab Blockchain Devnet",
+  "uid": "homelab-blockchain-devnet",
+  "version": 1,
+  "weekStart": ""
+}

bootstrap/platform/grafana-dashboards/homelab-capacity.json

@@ -0,0 +1,303 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Node Memory Used",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - (node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay|squashfs\",mountpoint=~\"/|/data|/var/lib/docker|/nvme-storage\"} / node_filesystem_size_bytes{fstype!~\"tmpfs|overlay|squashfs\",mountpoint=~\"/|/data|/var/lib/docker|/nvme-storage\"}))",
+          "legendFormat": "{{instance}} {{mountpoint}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Important Filesystems Used",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "sum by (node) (kube_pod_container_resource_requests{resource=\"cpu\"})",
+          "legendFormat": "{{node}} cpu cores requested",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node) (kube_pod_container_resource_limits{resource=\"cpu\"})",
+          "legendFormat": "{{node}} cpu cores limited",
+          "refId": "B"
+        }
+      ],
+      "title": "CPU Requests And Limits By Node",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "sum by (node) (kube_pod_container_resource_requests{resource=\"memory\"})",
+          "legendFormat": "{{node}} memory requested",
+          "refId": "A"
+        },
+        {
+          "expr": "sum by (node) (kube_pod_container_resource_limits{resource=\"memory\"})",
+          "legendFormat": "{{node}} memory limited",
+          "refId": "B"
+        }
+      ],
+      "title": "Memory Requests And Limits By Node",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "100 * (1 - kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes)",
+          "legendFormat": "{{namespace}}/{{persistentvolumeclaim}}",
+          "refId": "A"
+        }
+      ],
+      "title": "PVC Usage",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "homelab",
+    "jeannie",
+    "capacity"
+  ],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Homelab Capacity",
+  "uid": "homelab-capacity",
+  "version": 1,
+  "weekStart": ""
+}

bootstrap/platform/main.tf

@@ -107,6 +107,7 @@ EOT
 
   platform_topology_key          = "kubernetes.io/hostname"
   prometheus_stack_node_selector = var.prometheus_stack_node_selector
+  grafana_dashboard_files        = fileset("${path.module}/grafana-dashboards", "*.json")
   argocd_node_selector = {
     "kubernetes.io/os"           = "linux"
     "homelab.dev/workload-class" = "platform"
@@ -1496,6 +1497,15 @@ resource "helm_release" "prometheus_stack" {
       }
       grafana = {
         nodeSelector = local.prometheus_stack_node_selector
+        sidecar = {
+          dashboards = {
+            enabled          = true
+            label            = "grafana_dashboard"
+            labelValue       = "1"
+            searchNamespace  = var.observability.namespace
+            folderAnnotation = "grafana_folder"
+          }
+        }
         persistence = {
           enabled          = true
           type             = "sts"
@@ -1542,6 +1552,27 @@ resource "helm_release" "prometheus_stack" {
   ]
 }
 
+resource "kubernetes_config_map" "grafana_dashboards" {
+  for_each = local.grafana_dashboard_files
+
+  metadata {
+    name      = "homelab-grafana-${trimsuffix(each.key, ".json")}"
+    namespace = var.observability.namespace
+    labels = {
+      grafana_dashboard = "1"
+    }
+    annotations = {
+      grafana_folder = "Homelab"
+    }
+  }
+
+  data = {
+    (each.key) = file("${path.module}/grafana-dashboards/${each.key}")
+  }
+
+  depends_on = [helm_release.prometheus_stack]
+}
+
 resource "helm_release" "extra_tools" {
   for_each = var.extra_helm_releases
 

bootstrap/platform/variables.tf

@@ -251,5 +251,27 @@ variable "extra_helm_releases" {
     set_values       = map(string)
   }))
 
-  default = {}
+  default = {
+    tetragon = {
+      repository       = "https://helm.cilium.io"
+      chart            = "tetragon"
+      version          = ""
+      namespace        = "kube-system"
+      create_namespace = false
+      timeout          = 600
+      values_yaml      = <<-EOT
+        tetragon:
+          enableProcessCred: true
+          enableProcessNs: true
+          exportFilename: /var/run/cilium/tetragon/tetragon.log
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              memory: 512Mi
+      EOT
+      set_values       = {}
+    }
+  }
 }

docs/jeannie.1.md

@@ -23,8 +23,17 @@ The canonical script name is `metadata.main_script` in `homelab.yml`.
 
 ## COMMANDS
 
+Run `./jeannie help` for the short in-terminal command reference. It is grouped
+in the same order as the pipeline: inventory and planning, bootstrap, cluster
+lifecycle, reports, observability, recovery, doctors, access, apps, security,
+AI, and destructive commands.
+
 ### Core Lifecycle
 
+`help`
+: Print the grouped command reference. The same output is shown for `-h`,
+`--help`, or no command.
+
 `up`
 : Deploy the full homelab pipeline. Runs early preflight, deploys Gitea and RPi
 services, runs full preflight and pre-apply doctor checks, then applies Pimox,
@@ -48,7 +57,132 @@ cluster state.
 `status`
 : Print a cascade health report for the configured homelab, from bootstrap
 readiness through public website health. Set `LAB_STATUS_DETAILS=true` to append
-the older detailed tables.
+the older detailed tables. The cascade includes "what broke" signals for recent
+deployment readiness, pod restarts, node pressure, disk pressure, Traefik
+5xx/404 log evidence, and recent Gitea errors.
+
+`capacity`
+: Print a compact read-only placement report covering Debian, Kubernetes,
+Pimox, RPi, and placement signals. Use `capacity --details`, `--json`, `--only
+problems`, or `--verbose` for the older full dump.
+
+`recover-plan`
+: Print the ordered disaster recovery checklist and lightweight prerequisite
+checks for Debian, Gitea, RPi DNS, Pimox, Kubernetes, GitOps apps, and edge
+verification.
+
+`recover-power [--dry-run]`
+: Run the post-outage recovery sequence in dependency order: Debian runtime,
+Gitea, RPi DNS, Pimox workers, Kubernetes, GitOps/apps, and edge/public checks.
+
+`scorecard`
+: Roll up backup readiness, GitOps health, DNS/RPi health, cluster health, edge
+health, capacity pressure, security posture, public certificates, inventory, and
+docs freshness into a compact pass/warn/fail report.
+
+Report renderer
+: Report-oriented commands can share `scripts/report-render`, which reads status
+TSV rows and prints compact grouped summaries, detailed output, or JSON. Rows
+may include separate `fix`, `graph`, and `query` fields for Grafana-backed
+checks.
+
+`gitops-status`
+: Print a focused Argo CD status view with application sync/health, out-of-sync
+or degraded apps, repository secret presence, recent events, and recent
+repo-server/application-controller errors.
+
+`grafana-dashboards {list|apply}`
+: List or rebuild repo-provisioned Grafana dashboard ConfigMaps from
+`bootstrap/platform/grafana-dashboards`.
+
+`cert-check`
+: Verify public DNS, TLS certificate expiry, public website/Gitea HTTP status,
+and DuckDNS-to-OCI edge IP drift from the canonical inventory.
+
+`release-snapshot`
+: Write a timestamped pre-change report with Git state, Kubernetes/Argo CD/Helm
+state, workload images, public URL status, and latest Gitea/OpenTofu backup
+pointers.
+
+`backup-status`
+: Report freshness for Gitea backups, OpenTofu state backups, restore drill
+reports, and repo-managed Pi-hole restore inputs. The main `status` cascade
+includes this as a warning signal.
+
+`synthetic-checks`
+: Run end-to-end probes for Gitea HTTP/SSH, registry API, Pi-hole DNS, Traefik
+LAN HTTP, and temporary Kubernetes DNS/ingress pods. Registry push/pull is
+opt-in through `LAB_SYNTHETIC_REGISTRY_PUSH=true`.
+
+`resource-budget`
+: Check the report-only resource policy in `infra/resource-budgets.yml` against
+Kubernetes pod requests/limits and Debian disk/memory signals. Use
+`resource-budget --details` for the full pod/container list.
+
+`control-plane {status|taint|untaint|pods}`
+: Show or manage the Debian Kubernetes control-plane scheduling taint. Use
+`control-plane taint` after Pimox workers are stable to keep normal app/data
+plane pods off the Debian host.
+
+`artifact-cache {status|up|down|instructions}`
+: Manage the optional Debian-host artifact cache stack for apt packages and
+Docker Hub pull-through image caching.
+
+`blockchain-devnet {status|up|down|logs|rpc}`
+: Manage the optional local Ethereum Anvil devnet stack under
+`infra/blockchain-devnet`. This is for local learning only; never use real seed
+phrases or mainnet keys.
+
+`blockchain-test [forge args...]`
+: Run the Foundry smart contract tests in `labs/blockchain`. Extra arguments are
+passed through to `forge test`.
+
+`blockchain-wallet {instructions|new|address|sign-message MESSAGE}`
+: Practice dev wallet generation, address derivation, and message signing with
+Foundry Cast. Use disposable Anvil/dev keys only.
+
+`golden-ledger {show|check}`
+: Show or validate the Pimox golden image version ledger.
+
+`route-inventory`
+: Report Kubernetes ingress hosts, paths, backend services, TLS coverage, and
+visible Uptime Kuma monitor coverage.
+
+`workers <command>`
+: Manage Kubernetes/Pimox worker lifecycle operations. Commands include `list`,
+`start`, `stop`, `drain`, `uncordon`, `recreate-plan`, and `rebalance`.
+
+`change-journal {list|path}`
+: List local journal entries written before risky commands, or print the journal
+directory path.
+
+`map`
+: Print a dependency map from the canonical inventory. Use `map --dot` for a
+Graphviz DOT graph.
+
+Prometheus alerts
+: High-signal alert rules are managed as the `apps/homelab-alerts` GitOps app.
+They cover node readiness, restart storms, unavailable deployments, storage
+pressure, Traefik 5xx, and core service visibility.
+
+NetworkPolicies
+: Owned application namespaces include egress policies. The website can reach
+DNS, Redis, and the Debian Ollama endpoint; the security lab can reach DNS and
+external web targets while remaining isolated from cluster and LAN ranges.
+
+`validate`
+: Run non-mutating repository checks used by the GitOps promotion flow:
+inventory, generated docs, Bash syntax, shellcheck, YAML parsing, OpenTofu
+formatting, tailnet policy validation, and optional static security checks.
+
+`access-audit`
+: Run a read-only access inventory across SSH targets, Git/Gitea remotes,
+Tailscale ACL validation, Kubernetes RBAC, automation service accounts, and the
+Gitea runner.
+
+`kubeconfig-readonly`
+: Generate a separate local read-only kubeconfig from the repo-managed
+`homelab-access/homelab-readonly` ServiceAccount.
 
 `nuke`
 : Destroy Kubernetes state and Pimox worker VMs. Requires
@@ -78,9 +212,16 @@ ready.
 `backup-gitea`
 : Run a Gitea backup on the Debian host.
 
+`drill-restore`
+: Run a read-only restore checklist for Gitea backups, Pi-hole repo-managed
+config, OpenTofu state backup archives, and GitOps rebuild inputs.
+
 `drill-gitea-restore`
 : Run the Gitea restore drill workflow.
 
+`drill-pihole-restore`
+: Validate that repo-managed Pi-hole restore inputs exist and are non-empty.
+
 `install-gitea-runner [TOKEN]`
 : Install or repair the Gitea Actions runner. If `TOKEN` is omitted, the command
 uses the configured token source.
@@ -127,6 +268,59 @@ structure.
 : Check local AI helper prerequisites, knowledge index, and Ollama availability
 when the backstage helper is enabled.
 
+### Defensive Security
+
+`security-scan`
+: Run the default defensive scan set: Trivy repo/IaC scans plus OWASP ZAP
+baseline passive scans for targets in `security/targets.txt`.
+
+`security-prepare`
+: Validate Docker and report prerequisites, then pre-pull scanner images.
+
+`security-zap`
+: Run only OWASP ZAP baseline passive scans against public targets.
+
+`security-k8s`
+: Run kube-bench CIS checks against the Debian kubeadm host.
+
+`security-host`
+: Run a Lynis host audit on Debian and print one highest-risk finding to focus
+on first. This command is report-only.
+
+`security-trivy`
+: Run only Trivy filesystem and IaC/config scans.
+
+`security-secrets`
+: Run only gitleaks secret scanning against this repo.
+
+`security-nuclei`
+: Run low-rate nuclei HTTP exposure scans against public targets.
+
+`security-web`
+: Run quick HTTPS, status, and defensive-header checks against owned web
+targets.
+
+`security-logs`
+: Summarize recent Gitea and Traefik logs for common scanner paths, suspicious
+user agents, auth failures, and HTTP errors.
+
+`security-runtime`
+: Check the Tetragon runtime-detection DaemonSet and print the command for
+watching recent events.
+
+`security-attack-path [namespace] [workload]`
+: Run a read-only compromise-path drill for a Kubernetes workload. Defaults to
+the public website deployment.
+
+Kyverno hardening policies
+: `apps/supply-chain-policy` includes audit-mode checks for privileged pods,
+privilege escalation, hostPath use, resource requests/limits, and mutable image
+tags.
+
+`apps/security-lab`
+: GitOps-managed internal namespace for intentionally vulnerable practice
+targets. Use `kubectl port-forward`; do not expose it publicly.
+
 ### Maintenance
 
 `state-backup`
@@ -161,6 +355,17 @@ model.
 `HOMELAB_INVENTORY_FILE`
 : Path to the canonical non-secret inventory. Defaults to `./homelab.yml`.
 
+`LAB_STATUS_LOG_SINCE`
+: Log window for status log checks. Defaults to `30m`.
+
+`LAB_STATUS_RESTART_THRESHOLD`
+: Pod restart count that turns the status restart-pressure check into a warning.
+Defaults to `3`.
+
+`LAB_STATUS_DISK_USE_THRESHOLD`
+: Disk usage percentage that turns the status disk check into a warning.
+Defaults to `85`.
+
 `HOMELAB_STATE_DIR`
 : Local runtime state directory. Defaults to
 `${XDG_DATA_HOME:-$HOME/.local/share}/homelab`.
@@ -184,6 +389,35 @@ checks already show short output by default.
 : DNS name used by `status` and `doctor-preapply` for the Pi-hole resolver
 smoke test. Defaults to `cloudflare.com`.
 
+`SECURITY_REPORT_DIR`
+: Output directory for defensive security reports. Defaults to
+`$HOMELAB_STATE_DIR/security-reports`.
+
+`SECURITY_TARGETS_FILE`
+: Public target allowlist for ZAP and nuclei scans. Defaults to
+`security/targets.txt`.
+
+`SECURITY_ZAP_TARGET`
+: Optional single URL override for `security-zap`.
+
+`SECURITY_NUCLEI_TARGET`
+: Optional single URL override for `security-nuclei`.
+
+`SECURITY_WEB_TARGET`
+: Optional single URL override for `security-web`.
+
+`SECURITY_LOG_SINCE`
+: Log window for `security-logs`. Defaults to `24h`.
+
+`lynis`
+: `security-host` requires Lynis to be installed on the Debian host. Install it
+explicitly with `sudo apt install lynis`.
+
+`LAB_BACKSTAGE_BRAIN_ENABLED=true`
+: Allows `security-host` to ask the local Ollama/RAG helper to choose the single
+highest-risk Lynis finding. If the helper is disabled or unavailable, Jeannie
+falls back to the first Lynis warning, then the first suggestion.
+
 `LAB_AUTO_APPROVE=false`
 : Disable automatic OpenTofu approval for apply paths that support confirmation.
 
@@ -249,6 +483,12 @@ script name.
 `$HOMELAB_STATE_DIR/tofu-state-backups`
 : Local OpenTofu and generated state backups.
 
+`$HOMELAB_STATE_DIR/security-reports`
+: Defensive scan reports.
+
+`security/targets.txt`
+: Public target allowlist for defensive web scans.
+
 ## EXAMPLES
 
 Start a three-worker homelab:
@@ -269,6 +509,30 @@ Run focused edge diagnostics:
 ./jeannie doctor-edge
 ```
 
+Run the default defensive scan set:
+
+```sh
+./jeannie security-scan
+```
+
+Prepare scanner images before the first run:
+
+```sh
+./jeannie security-prepare
+```
+
+Run a ZAP baseline scan against one URL:
+
+```sh
+SECURITY_ZAP_TARGET=https://lab2025.duckdns.org/ ./jeannie security-zap
+```
+
+Run a report-only Lynis host audit:
+
+```sh
+./jeannie security-host
+```
+
 Run the blocking pre-apply doctor directly:
 
 ```sh

infra/artifact-cache/README.md

@@ -0,0 +1,21 @@
+# Homelab Artifact Cache
+
+This optional Debian-host stack caches repeated downloads during rebuilds.
+
+- `apt-cacher-ng` on port `3142` for Debian/Ubuntu apt packages
+- Docker registry pull-through cache on port `5001` for Docker Hub images
+
+It is intentionally separate from the main homelab pipeline. Start it when you
+want faster rebuilds and a little more tolerance for upstream outages.
+
+```bash
+cd infra/artifact-cache
+docker compose up -d
+```
+
+Client examples:
+
+```bash
+echo 'Acquire::http::Proxy "http://192.168.100.73:3142";' | sudo tee /etc/apt/apt.conf.d/01homelab-cache
+docker pull 192.168.100.73:5001/library/debian:bookworm
+```

infra/artifact-cache/docker-compose.yml

@@ -0,0 +1,21 @@
+services:
+  apt_cache:
+    image: sameersbn/apt-cacher-ng:3.7.4-20220421
+    container_name: homelab-apt-cache
+    restart: unless-stopped
+    ports:
+      - "${APT_CACHE_PORT:-3142}:3142"
+    volumes:
+      - "${ARTIFACT_CACHE_ROOT:-/data/artifact-cache}/apt-cacher-ng:/var/cache/apt-cacher-ng"
+
+  registry_cache:
+    image: registry:2
+    container_name: homelab-registry-cache
+    restart: unless-stopped
+    ports:
+      - "${REGISTRY_CACHE_PORT:-5001}:5000"
+    environment:
+      REGISTRY_PROXY_REMOTEURL: https://registry-1.docker.io
+      REGISTRY_STORAGE_DELETE_ENABLED: "true"
+    volumes:
+      - "${ARTIFACT_CACHE_ROOT:-/data/artifact-cache}/registry:/var/lib/registry"

infra/blockchain-devnet/README.md

@@ -0,0 +1,90 @@
+# Blockchain Devnet
+
+This stack runs a local Ethereum-compatible devnet for learning smart contracts,
+wallets, JSON-RPC, block production, and defensive RPC operations.
+
+It is intentionally local/dev-only. Do not use real wallets, seed phrases, or
+mainnet private keys here.
+
+## Commands
+
+```bash
+./jeannie blockchain-devnet status
+./jeannie blockchain-devnet up
+./jeannie blockchain-devnet logs
+./jeannie blockchain-devnet down
+```
+
+The default RPC endpoint is:
+
+```text
+http://127.0.0.1:8545
+```
+
+The lightweight local explorer is available at:
+
+```text
+http://127.0.0.1:8089
+```
+
+The local RPC gateway is available at:
+
+```text
+http://127.0.0.1:18545
+```
+
+Use the gateway when you want to practice RPC protection patterns such as method
+restriction, rate limiting, CORS headers, and request logging. Keep the raw
+Anvil RPC for direct local development.
+
+The Prometheus metrics endpoint is available at:
+
+```text
+http://127.0.0.1:9109/metrics
+```
+
+Metrics include:
+
+- `homelab_blockchain_rpc_up`
+- `homelab_blockchain_chain_id`
+- `homelab_blockchain_latest_block`
+- `homelab_blockchain_peer_count`
+- `homelab_blockchain_gas_price_wei`
+- `homelab_blockchain_rpc_latency_seconds`
+
+Use `prometheus-scrape.example.yml` as the scrape target shape. The repo also
+includes the `Homelab Blockchain Devnet` Grafana dashboard under
+`bootstrap/platform/grafana-dashboards`; rebuild dashboard ConfigMaps with:
+
+```bash
+./jeannie grafana-dashboards apply
+```
+
+From another LAN host, use the Debian host IP:
+
+```text
+http://192.168.100.73:8545
+```
+
+## Useful RPC Checks
+
+```bash
+curl -sS -X POST http://127.0.0.1:8545 \
+  -H 'content-type: application/json' \
+  --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'
+
+curl -sS -X POST http://127.0.0.1:8545 \
+  -H 'content-type: application/json' \
+  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
+```
+
+## Environment
+
+```bash
+BLOCKCHAIN_RPC_PORT=8545
+BLOCKCHAIN_RPC_GATEWAY_PORT=18545
+BLOCKCHAIN_EXPLORER_PORT=8089
+BLOCKCHAIN_METRICS_PORT=9109
+BLOCKCHAIN_CHAIN_ID=31337
+BLOCKCHAIN_BLOCK_TIME=2
+```

infra/blockchain-devnet/docker-compose.yml

@@ -0,0 +1,58 @@
+services:
+  anvil:
+    image: ghcr.io/foundry-rs/foundry:latest
+    container_name: homelab-blockchain-anvil
+    command:
+      - anvil
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8545"
+      - --chain-id
+      - "${BLOCKCHAIN_CHAIN_ID:-31337}"
+      - --block-time
+      - "${BLOCKCHAIN_BLOCK_TIME:-2}"
+      - --allow-origin
+      - "*"
+    ports:
+      - "${BLOCKCHAIN_RPC_PORT:-8545}:8545"
+    restart: unless-stopped
+
+  explorer:
+    image: nginx:1.27-alpine
+    container_name: homelab-blockchain-explorer
+    depends_on:
+      - anvil
+    ports:
+      - "${BLOCKCHAIN_EXPLORER_PORT:-8089}:80"
+    volumes:
+      - ./explorer:/usr/share/nginx/html:ro
+    restart: unless-stopped
+
+  rpc-gateway:
+    image: nginx:1.27-alpine
+    container_name: homelab-blockchain-rpc-gateway
+    depends_on:
+      - anvil
+    ports:
+      - "${BLOCKCHAIN_RPC_GATEWAY_PORT:-18545}:80"
+    volumes:
+      - ./nginx/rpc-gateway.conf:/etc/nginx/conf.d/default.conf:ro
+    restart: unless-stopped
+
+  metrics:
+    image: python:3.12-alpine
+    container_name: homelab-blockchain-metrics
+    depends_on:
+      - anvil
+    environment:
+      BLOCKCHAIN_METRICS_RPC_URL: http://anvil:8545
+      BLOCKCHAIN_METRICS_PORT: "9109"
+    ports:
+      - "${BLOCKCHAIN_METRICS_PORT:-9109}:9109"
+    volumes:
+      - ./metrics/exporter.py:/app/exporter.py:ro
+    command:
+      - python
+      - /app/exporter.py
+    restart: unless-stopped

infra/blockchain-devnet/explorer/index.html

@@ -0,0 +1,187 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Homelab Devnet Explorer</title>
+  <style>
+    :root {
+      color-scheme: light dark;
+      font-family: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+      line-height: 1.4;
+    }
+
+    body {
+      margin: 0;
+      background: #111827;
+      color: #f9fafb;
+    }
+
+    main {
+      width: min(980px, calc(100% - 32px));
+      margin: 0 auto;
+      padding: 32px 0;
+    }
+
+    h1 {
+      margin: 0 0 8px;
+      font-size: 1.8rem;
+    }
+
+    .toolbar,
+    .grid,
+    pre {
+      border: 1px solid #374151;
+      background: #1f2937;
+      border-radius: 8px;
+    }
+
+    .toolbar {
+      display: flex;
+      gap: 8px;
+      align-items: center;
+      padding: 12px;
+      margin: 24px 0;
+    }
+
+    input {
+      flex: 1;
+      min-width: 0;
+      padding: 10px;
+      border: 1px solid #4b5563;
+      border-radius: 6px;
+      background: #111827;
+      color: #f9fafb;
+    }
+
+    button {
+      padding: 10px 14px;
+      border: 0;
+      border-radius: 6px;
+      background: #38bdf8;
+      color: #0f172a;
+      font-weight: 700;
+      cursor: pointer;
+    }
+
+    .grid {
+      display: grid;
+      grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
+      gap: 1px;
+      overflow: hidden;
+    }
+
+    .metric {
+      padding: 16px;
+      background: #111827;
+    }
+
+    .metric span {
+      display: block;
+      color: #9ca3af;
+      font-size: 0.78rem;
+      text-transform: uppercase;
+    }
+
+    .metric strong {
+      display: block;
+      margin-top: 6px;
+      font-size: 1.2rem;
+      word-break: break-word;
+    }
+
+    pre {
+      padding: 16px;
+      white-space: pre-wrap;
+      word-break: break-word;
+      min-height: 120px;
+    }
+  </style>
+</head>
+<body>
+  <main>
+    <h1>Homelab Devnet Explorer</h1>
+    <p>Local Anvil chain view. Use only dev wallets and dev keys.</p>
+
+    <section class="toolbar">
+      <input id="rpcUrl" value="http://127.0.0.1:8545" aria-label="JSON-RPC URL">
+      <button id="refresh">Refresh</button>
+    </section>
+
+    <section class="grid" aria-label="Devnet metrics">
+      <div class="metric">
+        <span>Chain ID</span>
+        <strong id="chainId">unknown</strong>
+      </div>
+      <div class="metric">
+        <span>Latest Block</span>
+        <strong id="blockNumber">unknown</strong>
+      </div>
+      <div class="metric">
+        <span>Gas Price</span>
+        <strong id="gasPrice">unknown</strong>
+      </div>
+      <div class="metric">
+        <span>Peer Count</span>
+        <strong id="peerCount">unknown</strong>
+      </div>
+    </section>
+
+    <h2>Latest Block</h2>
+    <pre id="latestBlock">Waiting for refresh...</pre>
+  </main>
+
+  <script>
+    const fields = {
+      chainId: document.querySelector("#chainId"),
+      blockNumber: document.querySelector("#blockNumber"),
+      gasPrice: document.querySelector("#gasPrice"),
+      peerCount: document.querySelector("#peerCount"),
+      latestBlock: document.querySelector("#latestBlock"),
+      rpcUrl: document.querySelector("#rpcUrl")
+    };
+
+    async function rpc(method, params = []) {
+      const response = await fetch(fields.rpcUrl.value, {
+        method: "POST",
+        headers: { "content-type": "application/json" },
+        body: JSON.stringify({ jsonrpc: "2.0", method, params, id: Date.now() })
+      });
+      const payload = await response.json();
+      if (payload.error) {
+        throw new Error(payload.error.message || JSON.stringify(payload.error));
+      }
+      return payload.result;
+    }
+
+    function hexToDecimal(value) {
+      return Number.parseInt(value, 16).toString();
+    }
+
+    async function refresh() {
+      fields.latestBlock.textContent = "Loading...";
+      try {
+        const [chainId, blockNumber, gasPrice, peerCount] = await Promise.all([
+          rpc("eth_chainId"),
+          rpc("eth_blockNumber"),
+          rpc("eth_gasPrice"),
+          rpc("net_peerCount")
+        ]);
+        fields.chainId.textContent = hexToDecimal(chainId);
+        fields.blockNumber.textContent = hexToDecimal(blockNumber);
+        fields.gasPrice.textContent = `${hexToDecimal(gasPrice)} wei`;
+        fields.peerCount.textContent = hexToDecimal(peerCount);
+
+        const block = await rpc("eth_getBlockByNumber", [blockNumber, false]);
+        fields.latestBlock.textContent = JSON.stringify(block, null, 2);
+      } catch (error) {
+        fields.latestBlock.textContent = error.message;
+      }
+    }
+
+    document.querySelector("#refresh").addEventListener("click", refresh);
+    refresh();
+    setInterval(refresh, 10000);
+  </script>
+</body>
+</html>

infra/blockchain-devnet/metrics/exporter.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+import json
+import os
+import time
+import urllib.error
+import urllib.request
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+RPC_URL = os.environ.get("BLOCKCHAIN_METRICS_RPC_URL", "http://anvil:8545")
+LISTEN_ADDR = os.environ.get("BLOCKCHAIN_METRICS_ADDR", "0.0.0.0")
+LISTEN_PORT = int(os.environ.get("BLOCKCHAIN_METRICS_PORT", "9109"))
+
+
+def rpc(method):
+    started = time.monotonic()
+    body = json.dumps(
+        {"jsonrpc": "2.0", "method": method, "params": [], "id": 1}
+    ).encode("utf-8")
+    request = urllib.request.Request(
+        RPC_URL,
+        data=body,
+        headers={"content-type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(request, timeout=3) as response:
+        payload = json.loads(response.read().decode("utf-8"))
+    if "error" in payload:
+        raise RuntimeError(payload["error"])
+    return payload.get("result"), time.monotonic() - started
+
+
+def hex_to_int(value):
+    if isinstance(value, str) and value.startswith("0x"):
+        return int(value, 16)
+    return int(value)
+
+
+def collect_metrics():
+    metrics = []
+    latencies = {}
+    values = {}
+    up = 1
+
+    for method in ["eth_chainId", "eth_blockNumber", "net_peerCount", "eth_gasPrice"]:
+        try:
+            result, latency = rpc(method)
+            latencies[method] = latency
+            values[method] = hex_to_int(result)
+        except (OSError, RuntimeError, urllib.error.URLError, ValueError):
+            up = 0
+            latencies[method] = 0
+
+    metrics.append("# HELP homelab_blockchain_rpc_up Blockchain RPC health.")
+    metrics.append("# TYPE homelab_blockchain_rpc_up gauge")
+    metrics.append(f"homelab_blockchain_rpc_up {up}")
+
+    gauge_map = {
+        "eth_chainId": "homelab_blockchain_chain_id",
+        "eth_blockNumber": "homelab_blockchain_latest_block",
+        "net_peerCount": "homelab_blockchain_peer_count",
+        "eth_gasPrice": "homelab_blockchain_gas_price_wei",
+    }
+    for method, metric_name in gauge_map.items():
+        if method in values:
+            metrics.append(f"# TYPE {metric_name} gauge")
+            metrics.append(f"{metric_name} {values[method]}")
+
+    metrics.append("# TYPE homelab_blockchain_rpc_latency_seconds gauge")
+    for method, latency in latencies.items():
+        metrics.append(
+            f'homelab_blockchain_rpc_latency_seconds{{method="{method}"}} {latency:.6f}'
+        )
+
+    return "\n".join(metrics) + "\n"
+
+
+class MetricsHandler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        if self.path not in ["/metrics", "/"]:
+            self.send_response(404)
+            self.end_headers()
+            return
+
+        body = collect_metrics().encode("utf-8")
+        self.send_response(200)
+        self.send_header("content-type", "text/plain; version=0.0.4")
+        self.send_header("content-length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, fmt, *args):
+        return
+
+
+if __name__ == "__main__":
+    server = HTTPServer((LISTEN_ADDR, LISTEN_PORT), MetricsHandler)
+    server.serve_forever()

infra/blockchain-devnet/nginx/rpc-gateway.conf

@@ -0,0 +1,45 @@
+limit_req_zone $binary_remote_addr zone=rpc_per_ip:10m rate=10r/s;
+
+upstream anvil_rpc {
+    server anvil:8545;
+    keepalive 16;
+}
+
+server {
+    listen 80;
+    server_name _;
+
+    access_log /var/log/nginx/access.log;
+    error_log /var/log/nginx/error.log warn;
+
+    location = /healthz {
+        access_log off;
+        return 200 "ok\n";
+    }
+
+    location / {
+        if ($request_method = OPTIONS) {
+            add_header Access-Control-Allow-Origin "*" always;
+            add_header Access-Control-Allow-Methods "POST, OPTIONS" always;
+            add_header Access-Control-Allow-Headers "content-type" always;
+            add_header Access-Control-Max-Age 86400 always;
+            return 204;
+        }
+
+        limit_req zone=rpc_per_ip burst=20 nodelay;
+        limit_except POST {
+            deny all;
+        }
+
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_read_timeout 60s;
+        proxy_send_timeout 60s;
+
+        add_header Access-Control-Allow-Origin "*" always;
+        proxy_pass http://anvil_rpc;
+    }
+}

infra/blockchain-devnet/prometheus-scrape.example.yml

@@ -0,0 +1,6 @@
+- job_name: homelab-blockchain-devnet
+  static_configs:
+    - targets:
+        - 192.168.100.73:9109
+      labels:
+        service: blockchain-devnet

infra/pimox/golden-image-ledger.yml

@@ -0,0 +1,27 @@
+# Record the intended golden image inputs for Pimox worker templates.
+# Update this when rebuilding VM template 9000 or changing Kubernetes/runtime pins.
+template:
+  vmid: 9000
+  name: debian-13-arm64-worker-template
+  storage: local
+  worker_clone_storage: opi5_ssd
+os:
+  distribution: Debian
+  release: "13"
+  architecture: arm64
+  image_url: ""
+  image_sha256: ""
+kubernetes:
+  kubelet: v1.36
+  kubeadm: v1.36
+  kubectl: v1.36
+runtime:
+  containerd: ""
+  qemu_guest_agent: ""
+build:
+  last_built_utc: ""
+  built_by: ""
+  source_commit: ""
+notes:
+  - Treat Pimox worker disks as rebuildable capacity.
+  - Rebuild workers from the template instead of hand-upgrading drifted nodes.

infra/resource-budgets.yml

@@ -0,0 +1,29 @@
+# Report-only resource budgets for the current homelab hardware.
+# Tune these as the HP laptop, Pimox workers, and Raspberry Pi roles change.
+cluster:
+  max_total_cpu_millicores: 9000
+  max_total_memory_mib: 22000
+  require_requests: true
+  require_limits: true
+namespaces:
+  website-production:
+    max_cpu_millicores: 1000
+    max_memory_mib: 1024
+  demos-static:
+    max_cpu_millicores: 1000
+    max_memory_mib: 1024
+  argocd:
+    max_cpu_millicores: 2000
+    max_memory_mib: 3072
+  traefik:
+    max_cpu_millicores: 1000
+    max_memory_mib: 1024
+hosts:
+  debian:
+    min_root_free_percent: 15
+    min_data_free_percent: 20
+    max_memory_used_percent: 80
+  rpi4:
+    min_docker_free_percent: 15
+  pimox:
+    min_worker_storage_free_percent: 15

infra/rpi-services/README.md

@@ -11,6 +11,10 @@ Kubernetes cluster:
 
 Pi-hole also keeps public fallback upstreams in its config, so DNS can continue
 to resolve through `1.1.1.1` and `9.9.9.9` if the Unbound container stops.
+For Pi-hole v6, the stack sets `FTLCONF_dns_interface=ALL` and
+`FTLCONF_dns_listeningMode=ALL` through `PIHOLE_DNS_INTERFACE` and
+`PIHOLE_DNS_LISTENING_MODE` so LAN clients such as the Debian control host are
+not rejected as non-local networks.
 
 The bootstrap script prefers Docker data under `/nvme-storage/docker` only when
 that path is an active mount point and writable. If the NVMe is missing or
@@ -32,6 +36,7 @@ Useful overrides:
 ```bash
 LAB_RPI_HOST=192.168.100.89 ./jeannie rpi-services
 PIHOLE_WEBPASSWORD='<password>' ./jeannie rpi-services
+PIHOLE_DNS_INTERFACE=ALL PIHOLE_DNS_LISTENING_MODE=ALL ./jeannie rpi-services
 PIHOLE_WEB_PORT=8081 UPTIME_KUMA_PORT=3001 ./jeannie rpi-services
 LAB_RPI_STOP_LEGACY_PIHOLE=false ./jeannie rpi-services
 ```

infra/rpi-services/bootstrap.sh

@@ -140,6 +140,8 @@ PIHOLE_WEB_PORT=${PIHOLE_WEB_PORT:-8081}
 PIHOLE_DNS_BIND_IP=${PIHOLE_DNS_BIND_IP:-0.0.0.0}
 PIHOLE_WEB_BIND_IP=${PIHOLE_WEB_BIND_IP:-0.0.0.0}
 PIHOLE_WEBPASSWORD=${PIHOLE_WEBPASSWORD}
+PIHOLE_DNS_INTERFACE=${PIHOLE_DNS_INTERFACE:-ALL}
+PIHOLE_DNS_LISTENING_MODE=${PIHOLE_DNS_LISTENING_MODE:-ALL}
 PIHOLE_UPSTREAMS=${PIHOLE_UPSTREAMS:-unbound#53;1.1.1.1;9.9.9.9}
 PIHOLE_DATA_DIR=${PIHOLE_DATA_DIR:-${DATA_DIR}/pihole/etc-pihole}
 PIHOLE_DNSMASQ_DIR=${PIHOLE_DNSMASQ_DIR:-${DATA_DIR}/pihole/etc-dnsmasq.d}
@@ -253,9 +255,7 @@ for line in useful_lines(static_dhcp_hosts_file):
 PY
     sudo docker cp "${dnsmasq_file}" "${pihole_container}:/etc/dnsmasq.d/02-homelab-managed-dns.conf"
     rm -f "${dnsmasq_file}"
-    if ! sudo docker exec "${pihole_container}" pihole restartdns reload; then
-        sudo docker restart "${pihole_container}" >/dev/null
-    fi
+    sudo docker restart "${pihole_container}" >/dev/null
 }
 
 seed_uptime_kuma_monitors() {

infra/rpi-services/docker-compose.yml

@@ -22,6 +22,8 @@ services:
       TZ: ${TZ:-America/Mexico_City}
       WEBPASSWORD: ${PIHOLE_WEBPASSWORD:-homelab-change-me}
       DNSMASQ_LISTENING: all
+      FTLCONF_dns_interface: ${PIHOLE_DNS_INTERFACE:-ALL}
+      FTLCONF_dns_listeningMode: ${PIHOLE_DNS_LISTENING_MODE:-ALL}
       PIHOLE_DNS_: ${PIHOLE_UPSTREAMS:-unbound#53;1.1.1.1;9.9.9.9}
       FTLCONF_dns_upstreams: ${PIHOLE_UPSTREAMS:-unbound#53;1.1.1.1;9.9.9.9}
       REV_SERVER: ${PIHOLE_REV_SERVER:-false}

infra/rpi-services/seed-uptime-kuma.js

@@ -61,7 +61,7 @@ function loadDatabaseModule() {
       // Try the next location used by the upstream Uptime Kuma image.
     }
   }
-  throw new Error("Could not load better-sqlite3 from the Uptime Kuma container");
+  return null;
 }
 
 function nowSql() {
@@ -192,6 +192,10 @@ function main() {
   const databasePath = process.argv[3] || "/app/data/kuma.db";
   const monitors = JSON.parse(fs.readFileSync(monitorFile, "utf8"));
   const Database = loadDatabaseModule();
+  if (!Database) {
+    console.log("Could not load better-sqlite3 from the Uptime Kuma container; skipping monitor seed");
+    return;
+  }
   const database = new Database(databasePath);
 
   const userTable = database

jeannie

@@ -1487,6 +1487,7 @@ check_pihole_dns_query() {
     if command -v dig >/dev/null 2>&1; then
         output="$(dig @"${rpi_host}" "${query_name}" A +time=3 +tries=1 +short 2>&1)" || {
             printf '%s\n' "${output}" >&2
+            echo "Pi-hole DNS query to ${rpi_host}:53 failed; check RPi firewall and that the Pi-hole container publishes 53/udp on the LAN address." >&2
             return 1
         }
         if [[ -z "${output}" ]]; then
@@ -4657,12 +4658,10 @@ status_http_reachable() {
     esac
 }
 
-status_gitea_public_repo_ok() {
+status_gitea_public_ok() {
     local root_url="${LAB_GITEA_ROOT_URL:-${LAB_PUBLIC_URL:-https://lab2025.duckdns.org}/git/}"
-    local repo_owner="${LAB_GITEA_REPO_OWNER:-jv}"
-    local repo_name="${LAB_GITEA_REPO_NAME:-my-homelab-configs}"
 
-    status_http_ok "${root_url%/}/${repo_owner}/${repo_name}"
+    status_http_reachable "${root_url}"
 }
 
 status_cascade_check() {
@@ -4768,6 +4767,103 @@ status_no_problem_pods() {
     fi
 }
 
+status_recent_deployments() {
+    local rows
+
+    rows="$(
+        kubectl --kubeconfig "${KUBECONFIG_PATH}" get deployments -A \
+            -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.readyReplicas,DESIRED:.spec.replicas,UPDATED:.status.updatedReplicas,AVAILABLE:.status.availableReplicas,AGE:.metadata.creationTimestamp' \
+            --no-headers 2>/dev/null |
+            awk '$3 != $4 || $5 != $4 || $6 != $4 { print }'
+    )"
+    if [[ -n "${rows}" ]]; then
+        printf '%s\n' "${rows}" >&2
+        return 1
+    fi
+}
+
+status_pod_restart_pressure() {
+    local threshold="${LAB_STATUS_RESTART_THRESHOLD:-3}"
+    local rows
+
+    rows="$(
+        kubectl --kubeconfig "${KUBECONFIG_PATH}" get pods -A --no-headers 2>/dev/null |
+            awk -v threshold="${threshold}" '$5 + 0 >= threshold { print $1 "/" $2 ": restarts=" $5 " age=" $6 }'
+    )"
+    if [[ -n "${rows}" ]]; then
+        printf '%s\n' "${rows}" >&2
+        return 1
+    fi
+}
+
+status_node_pressure() {
+    local rows
+
+    rows="$(
+        kubectl --kubeconfig "${KUBECONFIG_PATH}" get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}{end}' 2>/dev/null |
+            awk '/MemoryPressure=True|DiskPressure=True|PIDPressure=True|NetworkUnavailable=True/ { print }'
+    )"
+    if [[ -n "${rows}" ]]; then
+        printf '%s\n' "${rows}" >&2
+        return 1
+    fi
+}
+
+status_disk_pressure() {
+    local threshold="${LAB_STATUS_DISK_USE_THRESHOLD:-85}"
+    local rows
+
+    rows="$(df -P / /data 2>/dev/null | awk -v threshold="${threshold}" 'NR > 1 { gsub(/%/, "", $5); if ($5 + 0 >= threshold) print $6 ": " $5 "% used" }')"
+    if [[ -n "${rows}" ]]; then
+        printf '%s\n' "${rows}" >&2
+        return 1
+    fi
+}
+
+status_traefik_error_rates() {
+    local since="${LAB_STATUS_LOG_SINCE:-30m}"
+    local sample
+    local count_502
+    local count_404
+
+    sample="$(kubectl --kubeconfig "${KUBECONFIG_PATH}" -n traefik logs -l app.kubernetes.io/name=traefik --since="${since}" --tail=2000 2>/dev/null || true)"
+    if [[ -z "${sample}" ]]; then
+        echo "no recent Traefik logs available" >&2
+        return 1
+    fi
+
+    count_502="$(grep -Eac '(^|[^0-9])50[234]([^0-9]|$)|level=error' <<<"${sample}" || true)"
+    count_404="$(grep -Eac '(^|[^0-9])404([^0-9]|$)' <<<"${sample}" || true)"
+    if ((count_502 > 0)); then
+        printf '5xx/error lines in last %s: %s\n' "${since}" "${count_502}" >&2
+        return 1
+    fi
+    printf '404 lines in last %s: %s\n' "${since}" "${count_404}" >&2
+}
+
+status_gitea_recent_errors() {
+    local since="${LAB_STATUS_LOG_SINCE:-30m}"
+    local container="${GITEA_CONTAINER_NAME:-homelab-gitea}"
+    local sample
+    local matches
+
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "docker not installed" >&2
+        return 1
+    fi
+    if ! sudo docker inspect "${container}" >/dev/null 2>&1; then
+        echo "container ${container} not found" >&2
+        return 1
+    fi
+
+    sample="$(sudo docker logs --since "${since}" "${container}" 2>&1 || true)"
+    matches="$(grep -Eai 'error|panic|fatal|authentication failed|denied| 50[0-9] ' <<<"${sample}" | tail -10 || true)"
+    if [[ -n "${matches}" ]]; then
+        printf '%s\n' "${matches}" >&2
+        return 1
+    fi
+}
+
 status_registry_ok() {
     status_http_ok "http://${LAB_REGISTRY_ENDPOINT:-192.168.100.73:30500}/v2/"
 }
@@ -4844,6 +4940,15 @@ status_cascade_report() {
     status_cascade_check "Kubernetes nodes Ready" status_kubernetes_nodes_ready || failures=$((failures + 1))
     status_cascade_warn "No problem pods" status_no_problem_pods
 
+    status_section "What Broke Signals"
+    status_cascade_warn "Backup freshness" backup_status
+    status_cascade_warn "Recent deployments healthy" status_recent_deployments
+    status_cascade_warn "Pod restart pressure" status_pod_restart_pressure
+    status_cascade_warn "Node pressure" status_node_pressure
+    status_cascade_warn "Disk usage" status_disk_pressure
+    status_cascade_warn "Traefik 5xx/404 signals" status_traefik_error_rates
+    status_cascade_warn "Gitea recent errors" status_gitea_recent_errors
+
     status_section "Platform And Apps"
     status_cascade_check "Local registry" status_registry_ok || failures=$((failures + 1))
     status_cascade_check "Traefik deployment" status_kubernetes_deployment_ready traefik traefik || failures=$((failures + 1))
@@ -4854,7 +4959,7 @@ status_cascade_report() {
     status_cascade_check "OCI edge SSH" check_edge_ssh || failures=$((failures + 1))
     status_cascade_check "Traefik LoadBalancer HTTP" status_http_reachable "http://${LAB_TRAEFIK_LB_IP:-192.168.100.240}/" || failures=$((failures + 1))
     status_cascade_check "Website public URL" status_http_ok "${LAB_PUBLIC_URL:-https://lab2025.duckdns.org}/" || failures=$((failures + 1))
-    status_cascade_check "Gitea public repo URL" status_gitea_public_repo_ok || failures=$((failures + 1))
+    status_cascade_check "Gitea public route" status_gitea_public_ok || failures=$((failures + 1))
 
     if ((failures > 0)); then
         printf '\nStatus cascade found %s blocking failure(s).\n' "${failures}" >&2
@@ -5761,26 +5866,420 @@ PY
     echo "AI check passed."
 }
 
+security_scan() {
+    require_debian_server "security-scan"
+    "${REPO_ROOT}/scripts/security-scan" all
+}
+
+security_prepare() {
+    require_debian_server "security-prepare"
+    "${REPO_ROOT}/scripts/security-scan" prepare
+}
+
+security_zap() {
+    require_debian_server "security-zap"
+    "${REPO_ROOT}/scripts/security-scan" zap
+}
+
+security_k8s() {
+    require_debian_server "security-k8s"
+    "${REPO_ROOT}/scripts/security-scan" kube-bench
+}
+
+security_host() {
+    require_debian_server "security-host"
+    "${REPO_ROOT}/scripts/security-scan" host
+}
+
+security_trivy() {
+    require_debian_server "security-trivy"
+    "${REPO_ROOT}/scripts/security-scan" trivy
+}
+
+security_secrets() {
+    require_debian_server "security-secrets"
+    "${REPO_ROOT}/scripts/security-scan" secrets
+}
+
+security_nuclei() {
+    require_debian_server "security-nuclei"
+    "${REPO_ROOT}/scripts/security-scan" nuclei
+}
+
+security_web() {
+    require_debian_server "security-web"
+    "${REPO_ROOT}/scripts/security-scan" web
+}
+
+security_logs() {
+    require_debian_server "security-logs"
+    "${REPO_ROOT}/scripts/security-logs"
+}
+
+security_runtime() {
+    require_debian_server "security-runtime"
+    if ! command -v kubectl >/dev/null 2>&1; then
+        echo "kubectl is required for security-runtime." >&2
+        exit 1
+    fi
+
+    kubectl --kubeconfig "${KUBECONFIG}" -n kube-system rollout status daemonset/tetragon --timeout=30s
+    kubectl --kubeconfig "${KUBECONFIG}" -n kube-system get pods -l app.kubernetes.io/name=tetragon -o wide
+    cat <<'EOF'
+
+Watch recent Tetragon events:
+  kubectl -n kube-system logs -l app.kubernetes.io/name=tetragon -c export-stdout --tail=100 -f
+
+Useful practice trigger:
+  kubectl -n security-lab exec deploy/juice-shop -- sh -c 'id; uname -a'
+EOF
+}
+
+security_attack_path() {
+    require_debian_server "security-attack-path"
+    "${REPO_ROOT}/scripts/security-attack-path" "${@:2}"
+}
+
+validate_homelab() {
+    "${REPO_ROOT}/scripts/validate-homelab"
+}
+
+access_audit() {
+    require_debian_server "access-audit"
+    "${REPO_ROOT}/scripts/access-audit"
+}
+
+drill_restore() {
+    require_debian_server "drill-restore"
+    "${REPO_ROOT}/scripts/restore-drill" all
+}
+
+drill_pihole_restore() {
+    "${REPO_ROOT}/scripts/restore-drill" pihole
+}
+
+kubeconfig_readonly() {
+    require_debian_server "kubeconfig-readonly"
+    "${REPO_ROOT}/scripts/kubeconfig-readonly"
+}
+
+capacity_report() {
+    require_debian_server "capacity"
+    "${REPO_ROOT}/scripts/capacity-report" "${@:2}"
+}
+
+recover_plan() {
+    require_debian_server "recover-plan"
+    "${REPO_ROOT}/scripts/recover-plan"
+}
+
+gitops_status() {
+    require_debian_server "gitops-status"
+    "${REPO_ROOT}/scripts/gitops-status"
+}
+
+cert_check() {
+    "${REPO_ROOT}/scripts/cert-check"
+}
+
+release_snapshot() {
+    require_debian_server "release-snapshot"
+    "${REPO_ROOT}/scripts/release-snapshot"
+}
+
+backup_status() {
+    require_debian_server "backup-status"
+    "${REPO_ROOT}/scripts/backup-status"
+}
+
+synthetic_checks() {
+    require_debian_server "synthetic-checks"
+    "${REPO_ROOT}/scripts/synthetic-checks"
+}
+
+resource_budget() {
+    require_debian_server "resource-budget"
+    "${REPO_ROOT}/scripts/resource-budget" "${@:2}"
+}
+
+control_plane() {
+    require_debian_server "control-plane"
+    "${REPO_ROOT}/scripts/control-plane" "${@:2}"
+}
+
+artifact_cache() {
+    require_debian_server "artifact-cache"
+    "${REPO_ROOT}/scripts/artifact-cache" "${@:2}"
+}
+
+blockchain_devnet() {
+    require_debian_server "blockchain-devnet"
+    "${REPO_ROOT}/scripts/blockchain-devnet" "${@:2}"
+}
+
+blockchain_test() {
+    "${REPO_ROOT}/scripts/blockchain-test" "${@:2}"
+}
+
+blockchain_wallet() {
+    "${REPO_ROOT}/scripts/blockchain-wallet" "${@:2}"
+}
+
+golden_ledger() {
+    "${REPO_ROOT}/scripts/golden-ledger" "${@:2}"
+}
+
+route_inventory() {
+    require_debian_server "route-inventory"
+    "${REPO_ROOT}/scripts/route-inventory"
+}
+
+recover_power() {
+    require_debian_server "recover-power"
+    record_change_journal "recover-power" "$@"
+    "${REPO_ROOT}/scripts/recover-power" "${@:2}"
+}
+
+scorecard() {
+    require_debian_server "scorecard"
+    "${REPO_ROOT}/scripts/scorecard"
+}
+
+grafana_dashboards() {
+    require_debian_server "grafana-dashboards"
+    "${REPO_ROOT}/scripts/grafana-dashboards" "${@:2}"
+}
+
+workers_manage() {
+    require_debian_server "workers"
+    "${REPO_ROOT}/scripts/workers" "${@:2}"
+}
+
+record_change_journal() {
+    local command_name="$1"
+    shift || true
+
+    "${REPO_ROOT}/scripts/change-journal" append "${command_name}" "$*" || true
+}
+
+change_journal() {
+    "${REPO_ROOT}/scripts/change-journal" "${@:2}"
+}
+
+homelab_map() {
+    "${REPO_ROOT}/scripts/homelab-map" "${@:2}"
+}
+
+print_usage() {
+    cat <<'EOF'
+Usage: ./jeannie <command> [args]
+
+Base, Inventory, And Planning
+  help | -h | --help                         Show this grouped help.
+  validate                                   Run repo validation checks.
+  inventory-check                            Validate the canonical homelab inventory.
+  preflight                                  Run non-mutating infrastructure preflight checks.
+  doctor-preapply                            Run deeper pre-apply safety checks.
+  plan [all|provisioning|cluster|platform|apps|edge]
+                                             Run OpenTofu plans without applying changes.
+
+Build And Bootstrap
+  up                                         Deploy the full homelab pipeline.
+  deploy-gitea                               Deploy the Debian-hosted Gitea container.
+  bootstrap-gitea-repo                       Ensure the Gitea repo and SSH key wiring exist.
+  rpi-services                               Deploy Pi-hole, Unbound, and Uptime Kuma on RPi4.
+  ollama-setup                               Install/configure Ollama on the Debian host.
+  artifact-cache {status|up|down|instructions}
+                                             Manage optional Debian artifact caches.
+  blockchain-devnet {status|up|down|logs|rpc}
+                                             Manage the local Ethereum Anvil devnet.
+  blockchain-test [forge args...]            Run Foundry tests for labs/blockchain.
+  blockchain-wallet {instructions|new|address|sign-message}
+                                             Practice dev wallet and signing workflows.
+  golden-ledger {show|check}                 Show or validate Pimox golden image versions.
+
+Cluster Lifecycle
+  rebuild-cluster                            Recreate the cluster through the guarded path.
+  stop-cluster                               Stop Kubernetes and worker VMs without destroy.
+  start-cluster                              Start Kubernetes and desired worker VMs.
+  workers <list|start|stop|drain|uncordon|recreate-plan|rebalance>
+                                             Manage Pimox/Kubernetes workers.
+  move-prometheus-stack-workers              Move monitoring workloads to worker nodes.
+  doctor-versions                            Check Kubernetes/container tooling versions.
+
+Operator View And Reports
+  status                                     Cascading health check from host to public URLs.
+  scorecard                                  Compact pass/warn/fail operator scorecard.
+  capacity                                   Compact capacity and placement report.
+  recover-plan                               Print disaster recovery order and prerequisites.
+  recover-power [--dry-run]                  Run or preview post-outage recovery.
+  map [--dot]                                Print the homelab dependency map.
+  change-journal {list|path}                 Show risky-command journal entries.
+  release-snapshot                           Write a pre-change release snapshot.
+
+Observability And GitOps
+  grafana-dashboards {list|apply}            List or apply repo-managed Grafana dashboards.
+  gitops-status                              Print focused Argo CD health and sync status.
+  cert-check                                 Check public DNS, TLS, and edge URL health.
+  backup-status                              Check backup and restore-drill freshness.
+  synthetic-checks                           Run end-to-end service probes.
+  resource-budget                            Report Kubernetes resource request/limit gaps.
+  route-inventory                            Report routes and Uptime Kuma coverage.
+  control-plane {status|taint|untaint|pods}
+                                             Keep normal pods off the Debian control plane.
+
+Recovery And State
+  backup-gitea                               Back up the Debian-hosted Gitea data.
+  drill-restore                              Run all restore drills.
+  drill-gitea-restore                        Run the Gitea restore drill.
+  drill-pihole-restore                       Run the Pi-hole restore drill.
+  state-backup                               Back up local OpenTofu state files.
+
+Focused Doctors
+  doctor-edge                                Diagnose edge routing and public exposure.
+  doctor-gitea                               Diagnose Gitea local/public access.
+  doctor-rpi                                 Diagnose RPi services and DNS.
+  doctor-cluster                             Diagnose Kubernetes cluster health.
+
+Access, Secrets, And Policy
+  access-audit                               Audit SSH, Gitea, Kubernetes, and Tailscale access.
+  kubeconfig-readonly                        Create/check read-only kubeconfig material.
+  secrets-init                               Initialize SOPS/age secret tooling.
+  secrets-check                              Validate repo-managed secret prerequisites.
+  tailnet-policy-check                       Validate Tailscale ACL policy as code.
+  fix-debian-docker-root                     Repair Debian Docker root placement.
+
+Apps And Services
+  apps                                       Deploy application stack only.
+  website-translation-model                  Prepare website translation model support.
+  website-ollama-listen                      Configure website Ollama access.
+  install-gitea-runner [TOKEN]               Install the Gitea Actions runner.
+  openwrt                                    Deploy/check OpenWrt lab config.
+
+Security Learning
+  security-scan                              Run the security scan bundle.
+  security-prepare                           Prepare defensive security tools.
+  security-zap                               Run OWASP ZAP checks.
+  security-k8s                               Run Kubernetes security checks.
+  security-host                              Run host security checks.
+  security-trivy                             Run Trivy scans.
+  security-secrets                           Run secret leak checks.
+  security-nuclei                            Run Nuclei web checks.
+  security-web                               Run web security checks.
+  security-logs                              Review security-relevant logs.
+  security-runtime                           Check runtime security sensors.
+  security-attack-path                       Print prioritized attack-path findings.
+
+AI And Indexing
+  ai-index                                   Build the local homelab RAG index.
+  ai-check                                   Query/check the local AI index.
+
+Destructive
+  nuke                                       Guarded cluster state destruction path.
+EOF
+}
+
 case "${1:-}" in
+    "" | help | -h | --help)
+        print_usage
+        ;;
     up)
+        record_change_journal "up" "$@"
         up
         ;;
     plan)
         plan_homelab "${2:-all}"
         ;;
     rebuild-cluster)
+        record_change_journal "rebuild-cluster" "$@"
         rebuild_cluster
         ;;
     stop-cluster)
+        record_change_journal "stop-cluster" "$@"
         stop_cluster
         ;;
     start-cluster)
+        record_change_journal "start-cluster" "$@"
         start_cluster
         ;;
     status)
         status_report
         ;;
+    capacity)
+        capacity_report "$@"
+        ;;
+    recover-plan)
+        recover_plan
+        ;;
+    gitops-status)
+        gitops_status
+        ;;
+    cert-check)
+        cert_check
+        ;;
+    release-snapshot)
+        record_change_journal "release-snapshot" "$@"
+        release_snapshot
+        ;;
+    backup-status)
+        backup_status
+        ;;
+    synthetic-checks)
+        synthetic_checks
+        ;;
+    resource-budget)
+        resource_budget "$@"
+        ;;
+    control-plane)
+        control_plane "$@"
+        ;;
+    artifact-cache)
+        artifact_cache "$@"
+        ;;
+    blockchain-devnet)
+        blockchain_devnet "$@"
+        ;;
+    blockchain-test)
+        blockchain_test "$@"
+        ;;
+    blockchain-wallet)
+        blockchain_wallet "$@"
+        ;;
+    golden-ledger)
+        golden_ledger "$@"
+        ;;
+    route-inventory)
+        route_inventory
+        ;;
+    recover-power)
+        recover_power "$@"
+        ;;
+    scorecard)
+        scorecard
+        ;;
+    grafana-dashboards)
+        grafana_dashboards "$@"
+        ;;
+    workers)
+        workers_manage "$@"
+        ;;
+    change-journal)
+        change_journal "$@"
+        ;;
+    map)
+        homelab_map "$@"
+        ;;
+    validate)
+        validate_homelab
+        ;;
+    access-audit)
+        access_audit
+        ;;
+    kubeconfig-readonly)
+        kubeconfig_readonly
+        ;;
     apps)
+        record_change_journal "apps" "$@"
         require_debian_server "apps"
         jeannie_log_start "apps"
         jeannie_step_plan 1
@@ -5797,9 +6296,11 @@ case "${1:-}" in
         ollama_setup
         ;;
     deploy-gitea)
+        record_change_journal "deploy-gitea" "$@"
         deploy_gitea
         ;;
     rpi-services)
+        record_change_journal "rpi-services" "$@"
         deploy_rpi_services
         ;;
     bootstrap-gitea-repo)
@@ -5808,9 +6309,15 @@ case "${1:-}" in
     backup-gitea)
         backup_gitea
         ;;
+    drill-restore)
+        drill_restore
+        ;;
     drill-gitea-restore)
         drill_gitea_restore
         ;;
+    drill-pihole-restore)
+        drill_pihole_restore
+        ;;
     install-gitea-runner)
         install_gitea_runner "${2:-}"
         ;;
@@ -5862,10 +6369,47 @@ case "${1:-}" in
     ai-check)
         ai_check
         ;;
+    security-scan)
+        security_scan
+        ;;
+    security-prepare)
+        security_prepare
+        ;;
+    security-zap)
+        security_zap
+        ;;
+    security-k8s)
+        security_k8s
+        ;;
+    security-host)
+        security_host
+        ;;
+    security-trivy)
+        security_trivy
+        ;;
+    security-secrets)
+        security_secrets
+        ;;
+    security-nuclei)
+        security_nuclei
+        ;;
+    security-web)
+        security_web
+        ;;
+    security-logs)
+        security_logs
+        ;;
+    security-runtime)
+        security_runtime
+        ;;
+    security-attack-path)
+        security_attack_path "$@"
+        ;;
     openwrt)
         openwrt
         ;;
     nuke)
+        record_change_journal "nuke" "$@"
         require_debian_server "nuke"
         jeannie_log_start "nuke"
         jeannie_step_plan 1
@@ -5873,7 +6417,8 @@ case "${1:-}" in
         echo "Log: ${JEANNIE_LOG_FILE}"
         ;;
     *)
-        echo "Usage: $0 {up|plan [all|provisioning|cluster|platform|apps|edge]|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|ollama-setup|deploy-gitea|rpi-services|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|doctor-edge|doctor-gitea|doctor-rpi|doctor-cluster|preflight|doctor-preapply|inventory-check|state-backup|fix-debian-docker-root|secrets-init|secrets-check|tailnet-policy-check|ai-index|ai-check|openwrt|nuke}"
+        printf 'Unknown command: %s\n\n' "$1" >&2
+        print_usage >&2
         exit 1
         ;;
 esac

labs/blockchain/README.md

@@ -0,0 +1,52 @@
+# Blockchain Contract Lab
+
+This is the local smart contract workspace for the homelab. It uses Foundry
+because it is fast, test-oriented, and works well with a local Anvil devnet.
+
+Never put real private keys, seed phrases, or funded wallet data in this repo.
+
+## Layout
+
+```text
+labs/blockchain/
+  foundry.toml
+  src/
+  test/
+  script/
+```
+
+## Test
+
+```bash
+./jeannie blockchain-test
+```
+
+The wrapper uses a local `forge` binary when available, otherwise it runs the
+Foundry Docker image with this directory mounted.
+
+## Deploy To The Local Devnet
+
+Start the local devnet:
+
+```bash
+./jeannie blockchain-devnet up
+```
+
+Deploy using one of Anvil's generated dev private keys:
+
+```bash
+cd labs/blockchain
+PRIVATE_KEY=0x... forge script script/DeployCounter.s.sol \
+  --rpc-url http://127.0.0.1:8545 \
+  --broadcast
+```
+
+Use only Anvil dev keys. Do not paste a real wallet key into a shell.
+
+## Security Exercises
+
+The intentionally vulnerable examples live in `SECURITY_LAB.md`. Start with:
+
+```bash
+./jeannie blockchain-test test/security/Reentrancy.t.sol -vv
+```

labs/blockchain/SECURITY_LAB.md

@@ -0,0 +1,42 @@
+# Smart Contract Security Lab
+
+This folder contains intentionally vulnerable contracts and tests that exploit
+them. The goal is to learn how bugs work so you can review and defend contracts
+more effectively.
+
+Never deploy these contracts outside a local devnet.
+
+## Current Exercises
+
+### Reentrancy
+
+Files:
+
+- `src/security/VulnerableBank.sol`
+- `test/security/Reentrancy.t.sol`
+
+The vulnerable contract sends ETH to the caller before updating internal
+balances. The attacker contract re-enters `withdraw` from its `receive`
+function and drains more ETH than it deposited.
+
+Run:
+
+```bash
+./jeannie blockchain-test test/security/Reentrancy.t.sol -vv
+```
+
+Defensive fixes to practice:
+
+- checks-effects-interactions
+- reentrancy guards
+- pull-payment patterns
+- avoiding unnecessary raw calls
+- invariant tests for total balances
+
+## Suggested Next Exercises
+
+- bad access control
+- `tx.origin` authentication
+- signature replay without chain ID or nonce
+- oracle price manipulation simulation
+- unsafe upgradeable proxy ownership

labs/blockchain/WALLET_LAB.md

@@ -0,0 +1,37 @@
+# Wallet And Signing Lab
+
+This lab is for disposable development wallets only.
+
+Do not use:
+
+- real seed phrases
+- hardware wallet recovery words
+- funded private keys
+- exchange account keys
+
+## Commands
+
+```bash
+./jeannie blockchain-wallet instructions
+./jeannie blockchain-wallet new
+export PRIVATE_KEY=0x...
+./jeannie blockchain-wallet address
+./jeannie blockchain-wallet sign-message "homelab devnet login"
+```
+
+## What To Learn
+
+- A private key controls an address.
+- A signature proves control of a private key without revealing it.
+- A transaction signature authorizes state change on a specific chain.
+- Chain IDs matter because they prevent replay across networks.
+- Seed phrases and private keys are secrets, not configuration.
+
+## Safe Practice Flow
+
+1. Start the devnet with `./jeannie blockchain-devnet up`.
+2. Use one of Anvil's printed dev private keys or generate a disposable key.
+3. Derive the address.
+4. Sign a harmless message.
+5. Verify the address and signature with Foundry/Cast experiments.
+6. Throw away the key when done.

labs/blockchain/foundry.toml

@@ -0,0 +1,12 @@
+[profile.default]
+src = "src"
+out = "out"
+libs = ["lib"]
+test = "test"
+script = "script"
+solc_version = "0.8.26"
+optimizer = true
+optimizer_runs = 200
+fs_permissions = [
+  { access = "read", path = "./" }
+]

labs/blockchain/script/DeployCounter.s.sol

@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.26;
+
+import {Script} from "forge-std/Script.sol";
+import {Counter} from "../src/Counter.sol";
+
+contract DeployCounter is Script {
+    function run() external returns (Counter counter) {
+        uint256 deployerKey = vm.envUint("PRIVATE_KEY");
+
+        vm.startBroadcast(deployerKey);
+        counter = new Counter(0);
+        vm.stopBroadcast();
+    }
+}

labs/blockchain/src/Counter.sol

@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.26;
+
+contract Counter {
+    uint256 public number;
+    address public owner;
+
+    error NotOwner();
+
+    event NumberChanged(uint256 previousNumber, uint256 newNumber);
+
+    constructor(uint256 initialNumber) {
+        owner = msg.sender;
+        number = initialNumber;
+    }
+
+    function setNumber(uint256 newNumber) external {
+        if (msg.sender != owner) {
+            revert NotOwner();
+        }
+
+        uint256 previousNumber = number;
+        number = newNumber;
+        emit NumberChanged(previousNumber, newNumber);
+    }
+
+    function increment() external {
+        uint256 previousNumber = number;
+        number++;
+        emit NumberChanged(previousNumber, number);
+    }
+}

labs/blockchain/src/security/VulnerableBank.sol

@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.26;
+
+contract VulnerableBank {
+    mapping(address account => uint256 balance) public balances;
+
+    function deposit() external payable {
+        balances[msg.sender] += msg.value;
+    }
+
+    function withdraw(uint256 amount) external {
+        require(balances[msg.sender] >= amount, "insufficient balance");
+
+        (bool ok,) = msg.sender.call{value: amount}("");
+        require(ok, "transfer failed");
+
+        unchecked {
+            balances[msg.sender] -= amount;
+        }
+    }
+
+    receive() external payable {}
+}

labs/blockchain/test/Counter.t.sol

@@ -0,0 +1,41 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.26;
+
+import {Test} from "forge-std/Test.sol";
+import {Counter} from "../src/Counter.sol";
+
+contract CounterTest is Test {
+    Counter private counter;
+    address private owner = address(0xA11CE);
+    address private stranger = address(0xB0B);
+
+    function setUp() public {
+        vm.prank(owner);
+        counter = new Counter(41);
+    }
+
+    function testInitialNumberAndOwner() public view {
+        assertEq(counter.number(), 41);
+        assertEq(counter.owner(), owner);
+    }
+
+    function testOwnerCanSetNumber() public {
+        vm.prank(owner);
+        counter.setNumber(100);
+
+        assertEq(counter.number(), 100);
+    }
+
+    function testNonOwnerCannotSetNumber() public {
+        vm.prank(stranger);
+        vm.expectRevert(Counter.NotOwner.selector);
+        counter.setNumber(100);
+    }
+
+    function testAnyoneCanIncrement() public {
+        vm.prank(stranger);
+        counter.increment();
+
+        assertEq(counter.number(), 42);
+    }
+}

labs/blockchain/test/security/Reentrancy.t.sol

@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.26;
+
+import {Test} from "forge-std/Test.sol";
+import {VulnerableBank} from "../../src/security/VulnerableBank.sol";
+
+contract ReentrancyAttacker {
+    VulnerableBank private immutable bank;
+    uint256 private immutable withdrawAmount;
+    uint256 public reentryCount;
+
+    constructor(VulnerableBank targetBank, uint256 targetWithdrawAmount) {
+        bank = targetBank;
+        withdrawAmount = targetWithdrawAmount;
+    }
+
+    function attack() external payable {
+        require(msg.value == withdrawAmount, "seed amount mismatch");
+
+        bank.deposit{value: msg.value}();
+        bank.withdraw(withdrawAmount);
+    }
+
+    receive() external payable {
+        if (address(bank).balance >= withdrawAmount && reentryCount < 3) {
+            reentryCount++;
+            bank.withdraw(withdrawAmount);
+        }
+    }
+}
+
+contract ReentrancyTest is Test {
+    VulnerableBank private bank;
+    ReentrancyAttacker private attacker;
+    address private victim = address(0xA11CE);
+
+    function setUp() public {
+        bank = new VulnerableBank();
+        attacker = new ReentrancyAttacker(bank, 1 ether);
+
+        vm.deal(victim, 5 ether);
+        vm.prank(victim);
+        bank.deposit{value: 5 ether}();
+    }
+
+    function testReentrancyDrainsMoreThanAttackerBalance() public {
+        attacker.attack{value: 1 ether}();
+
+        assertEq(attacker.reentryCount(), 3);
+        assertEq(address(attacker).balance, 4 ether);
+        assertEq(address(bank).balance, 2 ether);
+    }
+}

scripts/access-audit

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+KUBECONFIG="${KUBECONFIG:-${HOME}/.kube/config}"
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+warn_file_mode() {
+    local path="$1"
+    local mode
+
+    if [[ ! -e "${path}" ]]; then
+        printf '%-34s missing - %s\n' "key file" "${path}"
+        return 0
+    fi
+    mode="$(stat -c '%a' "${path}" 2>/dev/null || stat -f '%Lp' "${path}" 2>/dev/null || true)"
+    printf '%-34s %s mode=%s\n' "key file" "${path}" "${mode:-unknown}"
+    case "${mode}" in
+        600 | 400)
+            ;;
+        *)
+            printf '  warn: private SSH keys should normally be 0600 or 0400\n'
+            ;;
+    esac
+}
+
+ssh_inventory() {
+    section "SSH Inventory"
+    printf '%-18s %s@%s key=%s\n' "Debian" "${USER:-jv}" "${LAB_DEBIAN_HOST:-${LAB_GITEA_HOST:-192.168.100.73}}" "${LAB_DEBIAN_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
+    printf '%-18s %s@%s key=%s\n' "RPi4" "${LAB_RPI_USER:-${LAB_RASPBERRY_USER:-jv}}" "${LAB_RPI_HOST:-${LAB_RASPBERRY_HOST:-192.168.100.89}}" "${LAB_RPI_SSH_KEY_PATH:-${LAB_RASPBERRY_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}}"
+    printf '%-18s %s@%s key=%s\n' "Pimox" "${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}" "${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}" "${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-${HOME}/.ssh/id_ed25519}}"
+    printf '%-18s %s@%s\n' "OCI edge" "${LAB_EDGE_USER:-ubuntu}" "${LAB_EDGE_HOST:-132.145.170.74}"
+
+    warn_file_mode "${LAB_DEBIAN_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
+}
+
+git_access() {
+    section "Git Remotes"
+    git -C "${REPO_ROOT}" remote -v || true
+
+    printf '\nCurrent branch:\n'
+    git -C "${REPO_ROOT}" branch -vv || true
+
+    printf '\nGitea deploy key expected path: %s\n' "${LAB_GITEA_REPO_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519.pub}"
+    printf 'Gitea deploy key title:        %s\n' "${LAB_GITEA_REPO_DEPLOY_KEY_TITLE:-debian-host-my-homelab-configs}"
+    printf 'Gitea deploy key read-only:    %s\n' "${LAB_GITEA_REPO_DEPLOY_KEY_READ_ONLY:-false}"
+    if [[ "${LAB_GITEA_REPO_DEPLOY_KEY_READ_ONLY:-false}" != "true" ]]; then
+        printf '  warn: deploy key is expected to be write-capable for Actions/deploy mirroring\n'
+    fi
+}
+
+tailnet_access() {
+    section "Tailscale"
+    if command -v tailscale >/dev/null 2>&1; then
+        tailscale ip -4 || true
+        tailscale status || true
+    else
+        printf 'tailscale command not installed\n'
+    fi
+
+    printf '\nRepo-managed ACL validation:\n'
+    "${REPO_ROOT}/scripts/validate-tailnet-policy" || true
+}
+
+kubernetes_access() {
+    section "Kubernetes Access"
+    if ! command -v kubectl >/dev/null 2>&1; then
+        printf 'kubectl not installed\n'
+        return 0
+    fi
+    if [[ ! -s "${KUBECONFIG}" ]]; then
+        printf 'kubeconfig missing: %s\n' "${KUBECONFIG}"
+        return 0
+    fi
+
+    printf 'KUBECONFIG: %s\n' "${KUBECONFIG}"
+    kubectl --kubeconfig "${KUBECONFIG}" config current-context 2>/dev/null || true
+
+    printf '\nCan current user perform broad actions?\n'
+    for verb in get list create update delete; do
+        printf '%-10s ' "${verb}"
+        kubectl --kubeconfig "${KUBECONFIG}" auth can-i "${verb}" pods --all-namespaces 2>/dev/null || true
+    done
+
+    printf '\nClusterRoleBindings to cluster-admin:\n'
+    kubectl --kubeconfig "${KUBECONFIG}" get clusterrolebinding -o jsonpath='{range .items[?(@.roleRef.name=="cluster-admin")]}{.metadata.name}{" -> "}{range .subjects[*]}{.kind}{"/"}{.namespace}{"/"}{.name}{" "}{end}{"\n"}{end}' 2>/dev/null || true
+
+    printf '\nAutomation service accounts:\n'
+    kubectl --kubeconfig "${KUBECONFIG}" get serviceaccounts -A 2>/dev/null |
+        awk 'NR == 1 || /argocd|gitea|runner|kyverno|tetragon|prometheus|jeannie|homelab/'
+
+    cat <<'EOF'
+
+Read-only kubeconfig target:
+- Create a dedicated read-only ClusterRole/Binding for dashboards and audits.
+- Keep admin kubeconfig only on the Debian control host.
+- Use separate service accounts for automation instead of reusing admin.
+EOF
+}
+
+runner_access() {
+    section "Automation Runners"
+    if command -v systemctl >/dev/null 2>&1; then
+        systemctl is-active homelab-gitea-runner.service 2>/dev/null || true
+        systemctl status homelab-gitea-runner.service --no-pager -l 2>/dev/null | sed -n '1,12p' || true
+    else
+        printf 'systemctl unavailable\n'
+    fi
+}
+
+main() {
+    ssh_inventory
+    git_access
+    tailnet_access
+    kubernetes_access
+    runner_access
+}
+
+main "$@"

scripts/artifact-cache

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+STACK_DIR="${REPO_ROOT}/infra/artifact-cache"
+DEBIAN_IP="${LAB_DEBIAN_LAN_IP:-192.168.100.73}"
+APT_PORT="${APT_CACHE_PORT:-3142}"
+REGISTRY_PORT="${REGISTRY_CACHE_PORT:-5001}"
+
+case "${1:-status}" in
+    status)
+        echo "Artifact cache stack: ${STACK_DIR}"
+        if command -v docker >/dev/null 2>&1 && [ -d "$STACK_DIR" ]; then
+            (cd "$STACK_DIR" && docker compose ps) || true
+        else
+            echo "docker compose unavailable; cannot inspect local cache stack"
+        fi
+        ;;
+    up)
+        cd "$STACK_DIR"
+        docker compose up -d
+        ;;
+    down)
+        cd "$STACK_DIR"
+        docker compose down
+        ;;
+    instructions)
+        cat <<EOF
+APT proxy:
+  echo 'Acquire::http::Proxy "http://${DEBIAN_IP}:${APT_PORT}";' | sudo tee /etc/apt/apt.conf.d/01homelab-cache
+
+Docker Hub pull-through cache:
+  docker pull ${DEBIAN_IP}:${REGISTRY_PORT}/library/debian:bookworm
+
+Stack:
+  cd ${STACK_DIR}
+  docker compose up -d
+EOF
+        ;;
+    *)
+        echo "Usage: ./jeannie artifact-cache {status|up|down|instructions}" >&2
+        exit 1
+        ;;
+esac

scripts/backup-status

@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+GITEA_BACKUP_DIR="${LAB_GITEA_BACKUP_DIR:-/home/jv/backups/gitea}"
+GITEA_DRILL_DIR="${GITEA_RESTORE_DRILL_DIR:-/home/jv/backups/gitea-restore-drills}"
+TOFU_BACKUP_DIR="${HOMELAB_TOFU_STATE_BACKUP_DIR:-${HOMELAB_STATE_DIR}/tofu-state-backups}"
+PIHOLE_CONFIG_FILES="${PIHOLE_CONFIG_FILES:-infra/rpi-services/adlists.txt infra/rpi-services/local-dns-records.txt infra/rpi-services/cname-records.txt infra/rpi-services/static-dhcp-hosts.txt}"
+MAX_GITEA_AGE_HOURS="${LAB_BACKUP_MAX_GITEA_AGE_HOURS:-36}"
+MAX_TOFU_AGE_HOURS="${LAB_BACKUP_MAX_TOFU_AGE_HOURS:-168}"
+MAX_DRILL_AGE_HOURS="${LAB_BACKUP_MAX_DRILL_AGE_HOURS:-744}"
+
+failures=0
+warnings=0
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+latest_file() {
+    local dir="$1"
+    local pattern="$2"
+
+    if [ -d "$dir" ]; then
+        find "$dir" -maxdepth 1 -type f -name "$pattern" -printf '%T@ %p\n' 2>/dev/null |
+            sort -n |
+            tail -1 |
+            sed 's/^[^ ]* //'
+    fi
+}
+
+file_age_hours() {
+    local path="$1"
+    python3 - "$path" <<'PY'
+import os
+import sys
+import time
+
+path = sys.argv[1]
+age = int((time.time() - os.path.getmtime(path)) / 3600)
+print(age)
+PY
+}
+
+check_latest() {
+    local label="$1"
+    local dir="$2"
+    local pattern="$3"
+    local max_age_hours="$4"
+    local severity="${5:-fail}"
+    local latest
+    local age_hours
+
+    latest="$(latest_file "$dir" "$pattern" || true)"
+    printf '%-30s ' "$label"
+    if [ -z "$latest" ]; then
+        printf '%s - missing in %s\n' "$severity" "$dir"
+        if [ "$severity" = "fail" ]; then
+            failures=$((failures + 1))
+        else
+            warnings=$((warnings + 1))
+        fi
+        return 0
+    fi
+
+    age_hours="$(file_age_hours "$latest")"
+    if [ "$age_hours" -le "$max_age_hours" ]; then
+        printf 'ok - age=%sh path=%s\n' "$age_hours" "$latest"
+    else
+        printf '%s - age=%sh max=%sh path=%s\n' "$severity" "$age_hours" "$max_age_hours" "$latest"
+        if [ "$severity" = "fail" ]; then
+            failures=$((failures + 1))
+        else
+            warnings=$((warnings + 1))
+        fi
+    fi
+}
+
+section "Backup Freshness"
+check_latest "Gitea backup" "$GITEA_BACKUP_DIR" 'gitea-*.zip' "$MAX_GITEA_AGE_HOURS" fail
+check_latest "OpenTofu state backup" "$TOFU_BACKUP_DIR" 'tofu-state-*.tgz' "$MAX_TOFU_AGE_HOURS" warn
+check_latest "Restore drill report" "$GITEA_DRILL_DIR" 'restore-drill-*.txt' "$MAX_DRILL_AGE_HOURS" warn
+
+section "Repo Managed Pi-hole Inputs"
+for file in $PIHOLE_CONFIG_FILES; do
+    printf '%-30s ' "$file"
+    if [ -s "$file" ]; then
+        printf 'ok\n'
+    else
+        printf 'warn - missing or empty\n'
+        warnings=$((warnings + 1))
+    fi
+done
+
+section "Summary"
+printf 'failures=%s warnings=%s\n' "$failures" "$warnings"
+
+if [ "$failures" -gt 0 ]; then
+    exit 1
+fi

scripts/blockchain-devnet

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+STACK_DIR="${REPO_ROOT}/infra/blockchain-devnet"
+RPC_URL="${BLOCKCHAIN_RPC_URL:-http://127.0.0.1:${BLOCKCHAIN_RPC_PORT:-8545}}"
+RPC_GATEWAY_URL="${BLOCKCHAIN_RPC_GATEWAY_URL:-http://127.0.0.1:${BLOCKCHAIN_RPC_GATEWAY_PORT:-18545}}"
+
+usage() {
+    cat <<'EOF'
+Usage: ./jeannie blockchain-devnet {status|up|down|logs|rpc}
+
+status  Show local devnet containers and RPC health.
+up      Start the local Anvil devnet.
+down    Stop the local Anvil devnet.
+logs    Follow devnet logs.
+rpc     Print the configured JSON-RPC endpoint.
+EOF
+}
+
+rpc_call() {
+    local method="$1"
+
+    curl -fsS -X POST "$RPC_URL" \
+        -H 'content-type: application/json' \
+        --data "{\"jsonrpc\":\"2.0\",\"method\":\"${method}\",\"params\":[],\"id\":1}"
+}
+
+case "${1:-status}" in
+    status)
+        echo "Blockchain devnet stack: ${STACK_DIR}"
+        if command -v docker >/dev/null 2>&1; then
+            (cd "$STACK_DIR" && docker compose ps) || true
+        else
+            echo "docker compose unavailable; cannot inspect local devnet stack"
+        fi
+
+        echo
+        echo "RPC endpoint: ${RPC_URL}"
+        echo "RPC gateway: ${RPC_GATEWAY_URL}"
+        if rpc_call eth_chainId >/tmp/homelab-blockchain-chainid.json 2>/tmp/homelab-blockchain-rpc.err; then
+            printf 'chainId: '
+            cat /tmp/homelab-blockchain-chainid.json
+            echo
+            printf 'blockNumber: '
+            rpc_call eth_blockNumber || true
+            echo
+        else
+            echo "RPC unavailable"
+            sed 's/^/  /' /tmp/homelab-blockchain-rpc.err >&2 || true
+        fi
+        ;;
+    up)
+        cd "$STACK_DIR"
+        docker compose up -d
+        ;;
+    down)
+        cd "$STACK_DIR"
+        docker compose down
+        ;;
+    logs)
+        cd "$STACK_DIR"
+        docker compose logs -f --tail=100
+        ;;
+    rpc)
+        echo "raw=${RPC_URL}"
+        echo "gateway=${RPC_GATEWAY_URL}"
+        ;;
+    -h|--help|help)
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

scripts/blockchain-test

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+LAB_DIR="${REPO_ROOT}/labs/blockchain"
+FOUNDRY_IMAGE="${FOUNDRY_IMAGE:-ghcr.io/foundry-rs/foundry:latest}"
+
+usage() {
+    cat <<'EOF'
+Usage: ./jeannie blockchain-test [forge args...]
+
+Run Foundry tests for labs/blockchain. Extra arguments are passed to forge test.
+EOF
+}
+
+case "${1:-}" in
+    -h|--help|help)
+        usage
+        exit 0
+        ;;
+esac
+
+if command -v forge >/dev/null 2>&1; then
+    cd "$LAB_DIR"
+    exec forge test "$@"
+fi
+
+if ! command -v docker >/dev/null 2>&1; then
+    echo "forge or docker is required for blockchain-test." >&2
+    exit 1
+fi
+
+exec docker run --rm \
+    -v "${LAB_DIR}:/workspace" \
+    -w /workspace \
+    "$FOUNDRY_IMAGE" \
+    forge test "$@"

scripts/blockchain-wallet

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+FOUNDRY_IMAGE="${FOUNDRY_IMAGE:-ghcr.io/foundry-rs/foundry:latest}"
+
+usage() {
+    cat <<'EOF'
+Usage: ./jeannie blockchain-wallet {instructions|new|address|sign-message MESSAGE}
+
+instructions          Print wallet lab guidance.
+new                   Generate a new development wallet.
+address               Print the address for PRIVATE_KEY.
+sign-message MESSAGE  Sign a dev message with PRIVATE_KEY.
+
+Set PRIVATE_KEY only with an Anvil/dev private key. Never use a real wallet key.
+EOF
+}
+
+run_cast() {
+    if command -v cast >/dev/null 2>&1; then
+        cast "$@"
+        return
+    fi
+
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "cast or docker is required for blockchain-wallet." >&2
+        exit 1
+    fi
+
+    docker run --rm \
+        -e "PRIVATE_KEY=${PRIVATE_KEY:-}" \
+        "$FOUNDRY_IMAGE" \
+        cast "$@"
+}
+
+require_private_key() {
+    if [ -z "${PRIVATE_KEY:-}" ]; then
+        echo "Set PRIVATE_KEY to an Anvil/dev private key first." >&2
+        exit 1
+    fi
+}
+
+case "${1:-instructions}" in
+    instructions)
+        cat <<'EOF'
+Wallet lab rules:
+  - Use only Anvil or disposable development keys.
+  - Never paste a real seed phrase or funded private key into this repo or shell.
+  - Prefer environment variables for short-lived dev keys.
+  - Treat signatures as authorization; verify what you are signing.
+
+Examples:
+  ./jeannie blockchain-wallet new
+  export PRIVATE_KEY=0x...
+  ./jeannie blockchain-wallet address
+  ./jeannie blockchain-wallet sign-message "homelab devnet login"
+
+Pair this with:
+  ./jeannie blockchain-devnet up
+  ./jeannie blockchain-devnet rpc
+EOF
+        ;;
+    new)
+        run_cast wallet new
+        ;;
+    address)
+        require_private_key
+        run_cast wallet address "$PRIVATE_KEY"
+        ;;
+    sign-message)
+        require_private_key
+        shift
+        if [ "$#" -eq 0 ]; then
+            echo "sign-message requires a message." >&2
+            exit 1
+        fi
+        run_cast wallet sign --private-key "$PRIVATE_KEY" "$*"
+        ;;
+    -h|--help|help)
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

scripts/capacity-report

@@ -0,0 +1,399 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG="${KUBECONFIG:-${KUBECONFIG_PATH:-${HOME}/.kube/config}}"
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+GRAFANA_BASE_URL="${LAB_GRAFANA_URL:-https://grafana.${LAB_DOMAIN:-lab2025.duckdns.org}}"
+GRAFANA_CAPACITY_URL="${GRAFANA_BASE_URL%/}/d/homelab-capacity/homelab-capacity?orgId=1"
+GRAFANA_AVAILABILITY_URL="${GRAFANA_BASE_URL%/}/d/homelab-availability/homelab-availability?orgId=1"
+
+DETAIL_ARGS=()
+VERBOSE=false
+
+while (($# > 0)); do
+    case "$1" in
+        --verbose)
+            VERBOSE=true
+            shift
+            ;;
+        --details|--json)
+            DETAIL_ARGS+=("$1")
+            shift
+            ;;
+        --only)
+            DETAIL_ARGS+=("$1" "${2:-all}")
+            shift 2
+            ;;
+        -h|--help)
+            cat <<'EOF'
+Usage: ./jeannie capacity [--verbose] [--details] [--json] [--only failures|warnings|problems|all]
+
+Default output is a compact reusable report. Use --verbose for the older full
+capacity dump.
+EOF
+            exit 0
+            ;;
+        *)
+            echo "unknown capacity option: $1" >&2
+            exit 1
+            ;;
+    esac
+done
+
+emit() {
+    local status="$1"
+    local area="$2"
+    local check="$3"
+    local summary="$4"
+    local detail="${5:-}"
+    local fix="${6:-}"
+    local graph="${7:-}"
+    local query="${8:-}"
+
+    printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "${status}" "${area}" "${check}" "${summary}" "${detail}" "${fix}" "${graph}" "${query}"
+}
+
+grafana_explore_url() {
+    local expr="$1"
+    python3 - "${GRAFANA_BASE_URL%/}" "$expr" <<'PY'
+import json
+import sys
+import urllib.parse
+
+base_url, expr = sys.argv[1:3]
+left = {
+    "datasource": "prometheus",
+    "queries": [
+        {
+            "refId": "A",
+            "expr": expr,
+            "range": True,
+        }
+    ],
+    "range": {
+        "from": "now-6h",
+        "to": "now",
+    },
+}
+print(f"{base_url}/explore?left={urllib.parse.quote(json.dumps(left, separators=(',', ':')))}")
+PY
+}
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+run_optional() {
+    local label="$1"
+    shift
+
+    printf '\n-- %s\n' "${label}"
+    "$@" 2>&1 || true
+}
+
+host_capacity() {
+    section "Debian Host"
+    run_optional "memory" free -h
+    run_optional "disk" df -h / /data /var/lib/docker
+    if command -v docker >/dev/null 2>&1; then
+        run_optional "docker disk usage" sudo docker system df
+        run_optional "largest docker volumes" sudo docker system df -v
+    else
+        printf 'docker not installed\n'
+    fi
+}
+
+kubernetes_capacity() {
+    section "Kubernetes"
+    if ! command -v kubectl >/dev/null 2>&1 || [[ ! -s "${KUBECONFIG}" ]]; then
+        printf 'kubectl or kubeconfig unavailable\n'
+        return 0
+    fi
+
+    run_optional "nodes" kubectl --kubeconfig "${KUBECONFIG}" get nodes -o wide
+    run_optional "node resource summary" kubectl --kubeconfig "${KUBECONFIG}" describe nodes
+    run_optional "top nodes" kubectl --kubeconfig "${KUBECONFIG}" top nodes
+    run_optional "top pods" kubectl --kubeconfig "${KUBECONFIG}" top pods -A --sort-by=memory
+    run_optional "PVCs" kubectl --kubeconfig "${KUBECONFIG}" get pvc -A -o wide
+    run_optional "PVs" kubectl --kubeconfig "${KUBECONFIG}" get pv -o wide
+
+    printf '\n-- pods without resource requests or limits\n'
+    kubectl --kubeconfig "${KUBECONFIG}" get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{":cpuReq="}{.resources.requests.cpu}{",memReq="}{.resources.requests.memory}{",memLimit="}{.resources.limits.memory}{" "}{end}{"\n"}{end}' 2>/dev/null |
+        awk '/cpuReq=,|memReq=,|memLimit=,/ { print }' || true
+}
+
+pimox_capacity() {
+    local pimox_host="${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}"
+    local pimox_user="${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}"
+    local pimox_key="${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-${HOME}/.ssh/id_ed25519}}"
+
+    section "Pimox"
+    ssh -i "${pimox_key}" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${pimox_user}@${pimox_host}" '
+set +e
+echo "-- host memory"
+free -h
+echo
+echo "-- storage"
+df -h / /data 2>/dev/null || df -h
+echo
+echo "-- pvesm"
+sudo pvesm status 2>/dev/null || true
+echo
+echo "-- VMs"
+sudo qm list 2>/dev/null || true
+echo
+echo "-- running VM configs"
+for vmid in $(sudo qm list 2>/dev/null | awk "NR > 1 {print \$1}"); do
+  echo "VM ${vmid}"
+  sudo qm config "${vmid}" 2>/dev/null | awk "/^(memory|cores|sockets|scsi|virtio|sata|ide)[0-9]*:|^memory:|^cores:|^sockets:/"
+done
+' || printf 'unable to reach Pimox host %s@%s\n' "${pimox_user}" "${pimox_host}"
+}
+
+rpi_capacity() {
+    local rpi_host="${LAB_RPI_HOST:-${LAB_RASPBERRY_HOST:-192.168.100.89}}"
+    local rpi_user="${LAB_RPI_USER:-${LAB_RASPBERRY_USER:-jv}}"
+    local rpi_key="${LAB_RPI_SSH_KEY_PATH:-${LAB_RASPBERRY_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}}"
+
+    section "RPi4"
+    ssh -i "${rpi_key}" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${rpi_user}@${rpi_host}" '
+set +e
+echo "-- memory"
+free -h
+echo
+echo "-- disk"
+df -h / /nvme-storage /var/lib/docker 2>/dev/null || df -h
+echo
+echo "-- docker root"
+sudo docker info --format "{{.DockerRootDir}}" 2>/dev/null || true
+echo
+echo "-- docker disk usage"
+sudo docker system df 2>/dev/null || true
+' || printf 'unable to reach RPi host %s@%s\n' "${rpi_user}" "${rpi_host}"
+}
+
+summary() {
+    section "Capacity Guidance"
+    cat <<'EOF'
+Use this report to decide placement:
+- Prefer Kubernetes app workloads on Pimox workers with SSD-backed storage.
+- Keep Debian control-plane headroom for Docker, Gitea, registry, Ollama, and kubeadm.
+- Treat Orange Pi VM disks as rebuildable capacity, not durable backup storage.
+- Keep RPi4 focused on DNS/Uptime Kuma/light services until NVMe stability is proven.
+- Add capacity only when CPU, memory, and disk all have margin; storage alone is not enough.
+EOF
+}
+
+verbose_main() {
+    host_capacity
+    kubernetes_capacity
+    pimox_capacity
+    rpi_capacity
+    summary
+}
+
+compact_host_capacity() {
+    local mem_total
+    local mem_used
+    local mem_pct
+    local path
+    local line
+    local use_pct
+    local docker_root
+
+    if command -v free >/dev/null 2>&1; then
+        read -r mem_total mem_used < <(free -m | awk '/^Mem:/ { print $2, $3 }')
+        if [[ -n "${mem_total:-}" && "${mem_total}" -gt 0 ]]; then
+            mem_pct=$((mem_used * 100 / mem_total))
+            if ((mem_pct >= 85)); then
+                emit warn "Debian" "Memory" "used=${mem_pct}%" "${mem_used}MiB/${mem_total}MiB" "Move workloads to Pimox workers or stop nonessential containers." "${GRAFANA_CAPACITY_URL}&viewPanel=1" "$(grafana_explore_url '100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))')"
+            else
+                emit ok "Debian" "Memory" "used=${mem_pct}%" "${mem_used}MiB/${mem_total}MiB"
+            fi
+        else
+            emit skip "Debian" "Memory" "unavailable" "free output could not be parsed"
+        fi
+    else
+        emit skip "Debian" "Memory" "free missing" "" "Install procps."
+    fi
+
+    for path in / /data /var/lib/docker; do
+        line="$(df -P "${path}" 2>/dev/null | awk 'NR == 2 { print $5 " " $4 }' || true)"
+        if [[ -z "${line}" ]]; then
+            emit skip "Debian" "Disk ${path}" "unavailable" "path not present"
+            continue
+        fi
+        use_pct="${line%% *}"
+        use_pct="${use_pct%%%}"
+        if ((use_pct >= 90)); then
+            emit fail "Debian" "Disk ${path}" "used=${use_pct}%" "available blocks: ${line#* }" "Free disk or move data before deploying more services." "${GRAFANA_CAPACITY_URL}&viewPanel=2" "$(grafana_explore_url '100 * (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/data|/var/lib/docker|/nvme-storage"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/data|/var/lib/docker|/nvme-storage"}))')"
+        elif ((use_pct >= 80)); then
+            emit warn "Debian" "Disk ${path}" "used=${use_pct}%" "available blocks: ${line#* }" "Review Docker volumes, backups, and registry usage." "${GRAFANA_CAPACITY_URL}&viewPanel=2" "$(grafana_explore_url '100 * (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/data|/var/lib/docker|/nvme-storage"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/data|/var/lib/docker|/nvme-storage"}))')"
+        else
+            emit ok "Debian" "Disk ${path}" "used=${use_pct}%" "available blocks: ${line#* }"
+        fi
+    done
+
+    if command -v docker >/dev/null 2>&1; then
+        docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)"
+        if [[ -n "${docker_root}" ]]; then
+            emit ok "Debian" "Docker root" "${docker_root}" "" "Keep Docker on NVMe-backed storage."
+        else
+            emit warn "Debian" "Docker root" "unavailable" "docker info failed" "Check docker service."
+        fi
+    else
+        emit skip "Debian" "Docker" "not installed"
+    fi
+}
+
+compact_kubernetes_capacity() {
+    local not_ready
+    local problem_pods
+    local missing_resources
+    local top_nodes
+    local node_ready_query
+    local problem_pods_query
+    local resource_policy_query
+    local node_pressure_query
+
+    node_ready_query="$(grafana_explore_url 'kube_node_status_condition{condition="Ready",status="true"} == 0')"
+    problem_pods_query="$(grafana_explore_url 'sum by (namespace, pod, phase) (kube_pod_status_phase{phase!~"Running|Succeeded"} == 1)')"
+    resource_policy_query="$(grafana_explore_url 'sum by (namespace, pod) (kube_pod_container_resource_requests{resource=~"cpu|memory"})')"
+    node_pressure_query="$(grafana_explore_url '100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))')"
+
+    if ! command -v kubectl >/dev/null 2>&1 || [[ ! -s "${KUBECONFIG}" ]]; then
+        emit warn "Kubernetes" "API" "unavailable" "kubectl or kubeconfig missing" "Start the cluster or set KUBECONFIG."
+        return 0
+    fi
+    if ! kubectl --kubeconfig "${KUBECONFIG}" get --raw=/readyz >/dev/null 2>&1; then
+        emit warn "Kubernetes" "API" "not reachable" "readyz failed" "./jeannie start-cluster"
+        return 0
+    fi
+
+    not_ready="$(kubectl --kubeconfig "${KUBECONFIG}" get nodes --no-headers 2>/dev/null | awk '$2 !~ /Ready/ { count++ } END { print count + 0 }')"
+    if ((not_ready > 0)); then
+        emit fail "Kubernetes" "Node readiness" "${not_ready} not ready" "" "./jeannie doctor-cluster" "${GRAFANA_AVAILABILITY_URL}&viewPanel=1" "${node_ready_query}"
+    else
+        emit ok "Kubernetes" "Node readiness" "all ready"
+    fi
+
+    problem_pods="$(kubectl --kubeconfig "${KUBECONFIG}" get pods -A --no-headers 2>/dev/null | awk '$4 != "Running" && $4 != "Completed" { count++ } END { print count + 0 }')"
+    if ((problem_pods > 0)); then
+        emit warn "Kubernetes" "Problem pods" "${problem_pods}" "pods not Running/Completed" "kubectl get pods -A -o wide" "${GRAFANA_AVAILABILITY_URL}&viewPanel=2" "${problem_pods_query}"
+    else
+        emit ok "Kubernetes" "Problem pods" "none"
+    fi
+
+    missing_resources="$(kubectl --kubeconfig "${KUBECONFIG}" get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{":cpuReq="}{.resources.requests.cpu}{",memReq="}{.resources.requests.memory}{",memLimit="}{.resources.limits.memory}{" "}{end}{"\n"}{end}' 2>/dev/null |
+        awk '/cpuReq=,|memReq=,|memLimit=,/ { count++ } END { print count + 0 }')"
+    if ((missing_resources > 0)); then
+        emit warn "Kubernetes" "Resource policy" "${missing_resources} pods missing requests/limits" "" "./jeannie resource-budget" "${GRAFANA_CAPACITY_URL}&viewPanel=3" "${resource_policy_query}"
+    else
+        emit ok "Kubernetes" "Resource policy" "requests/limits present"
+    fi
+
+    top_nodes="$(kubectl --kubeconfig "${KUBECONFIG}" top nodes 2>/dev/null | awk 'NR > 1 { print $1 ":cpu=" $3 ",mem=" $5 }' | paste -sd ';' - || true)"
+    if [[ -n "${top_nodes}" ]]; then
+        emit ok "Kubernetes" "Node pressure" "metrics available" "${top_nodes}"
+    else
+        emit skip "Kubernetes" "Node pressure" "metrics unavailable" "metrics-server may not be installed or ready" "" "${GRAFANA_CAPACITY_URL}&viewPanel=1" "${node_pressure_query}"
+    fi
+}
+
+compact_pimox_capacity() {
+    local pimox_host="${LAB_PIMOX_HOST:-${TF_VAR_pimox_host:-192.168.100.80}}"
+    local pimox_user="${LAB_PIMOX_USER:-${TF_VAR_pimox_user:-jv}}"
+    local pimox_key="${LAB_PIMOX_SSH_KEY_PATH:-${TF_VAR_pimox_ssh_key_path:-${HOME}/.ssh/id_ed25519}}"
+    local worker_storage="${LAB_PIMOX_WORKER_STORAGE:-${TF_VAR_pimox_worker_storage:-opi5_ssd}}"
+    local output
+
+    output="$(ssh -i "${pimox_key}" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${pimox_user}@${pimox_host}" "set +e
+storage_line=\"\$(sudo pvesm status 2>/dev/null | awk -v storage='${worker_storage}' '\$1 == storage { print \$7 }')\"
+running_vms=\"\$(sudo qm list 2>/dev/null | awk 'NR > 1 && \$3 == \"running\" { count++ } END { print count + 0 }')\"
+total_vms=\"\$(sudo qm list 2>/dev/null | awk 'NR > 1 { count++ } END { print count + 0 }')\"
+echo \"storage_used_pct=\${storage_line:-unknown}\"
+echo \"running_vms=\${running_vms:-0}\"
+echo \"total_vms=\${total_vms:-0}\"
+" 2>/dev/null || true)"
+
+    if [[ -z "${output}" ]]; then
+        emit warn "Pimox" "Host" "unreachable" "${pimox_user}@${pimox_host}" "./jeannie doctor-cluster"
+        return 0
+    fi
+
+    local storage_pct
+    local running_vms
+    local total_vms
+    storage_pct="$(printf '%s\n' "${output}" | awk -F= '/storage_used_pct=/ { print $2 }')"
+    running_vms="$(printf '%s\n' "${output}" | awk -F= '/running_vms=/ { print $2 }')"
+    total_vms="$(printf '%s\n' "${output}" | awk -F= '/total_vms=/ { print $2 }')"
+
+    if [[ "${storage_pct}" =~ ^[0-9.]+%?$ ]]; then
+        storage_pct="${storage_pct%%%}"
+        if ((${storage_pct%.*} >= 85)); then
+            emit warn "Pimox" "Worker storage" "used=${storage_pct}%" "${worker_storage}" "Clean old VM disks or add storage."
+        else
+            emit ok "Pimox" "Worker storage" "used=${storage_pct}%" "${worker_storage}"
+        fi
+    else
+        emit warn "Pimox" "Worker storage" "unknown" "${worker_storage}" "Check pvesm status."
+    fi
+    emit ok "Pimox" "VMs" "running=${running_vms:-0}/${total_vms:-0}"
+}
+
+compact_rpi_capacity() {
+    local rpi_host="${LAB_RPI_HOST:-${LAB_RASPBERRY_HOST:-192.168.100.89}}"
+    local rpi_user="${LAB_RPI_USER:-${LAB_RASPBERRY_USER:-jv}}"
+    local rpi_key="${LAB_RPI_SSH_KEY_PATH:-${LAB_RASPBERRY_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}}"
+    local output
+
+    output="$(ssh -i "${rpi_key}" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${rpi_user}@${rpi_host}" '
+set +e
+docker_root="$(sudo docker info --format "{{.DockerRootDir}}" 2>/dev/null || true)"
+root_used="$(df -P / 2>/dev/null | awk "NR == 2 { print \$5 }")"
+nvme_used="$(df -P /nvme-storage 2>/dev/null | awk "NR == 2 { print \$5 }")"
+echo "docker_root=${docker_root:-unknown}"
+echo "root_used=${root_used:-unknown}"
+echo "nvme_used=${nvme_used:-unknown}"
+' 2>/dev/null || true)"
+
+    if [[ -z "${output}" ]]; then
+        emit warn "RPi4" "Host" "unreachable" "${rpi_user}@${rpi_host}" "./jeannie doctor-rpi"
+        return 0
+    fi
+
+    local docker_root
+    local root_used
+    local nvme_used
+    docker_root="$(printf '%s\n' "${output}" | awk -F= '/docker_root=/ { print $2 }')"
+    root_used="$(printf '%s\n' "${output}" | awk -F= '/root_used=/ { print $2 }')"
+    nvme_used="$(printf '%s\n' "${output}" | awk -F= '/nvme_used=/ { print $2 }')"
+    emit ok "RPi4" "Docker root" "${docker_root:-unknown}"
+    emit ok "RPi4" "Root disk" "${root_used:-unknown}"
+    if [[ "${nvme_used:-unknown}" = "unknown" ]]; then
+        emit warn "RPi4" "NVMe mount" "unknown" "df /nvme-storage failed" "./jeannie doctor-rpi"
+    else
+        emit ok "RPi4" "NVMe mount" "${nvme_used}"
+    fi
+}
+
+compact_guidance() {
+    emit ok "Guidance" "Placement" "Prefer app workloads on Pimox workers" "Keep Debian headroom for Gitea, registry, Docker, kubeadm, and Ollama."
+    emit ok "Guidance" "Storage" "Treat Pimox VM disks as rebuildable" "Durable data still needs explicit backup/restore paths."
+}
+
+compact_main() {
+    {
+        compact_host_capacity
+        compact_kubernetes_capacity
+        compact_pimox_capacity
+        compact_rpi_capacity
+        compact_guidance
+    } | "${REPO_ROOT}/scripts/report-render" --title "Homelab Capacity" "${DETAIL_ARGS[@]}"
+}
+
+if [[ "${VERBOSE}" == "true" ]]; then
+    verbose_main
+else
+    compact_main
+fi

scripts/cert-check

@@ -0,0 +1,168 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+INVENTORY_FILE="${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}"
+MIN_DAYS="${LAB_CERT_MIN_DAYS:-21}"
+DEFAULT_DOMAIN="${LAB_DOMAIN:-lab2025.duckdns.org}"
+DEFAULT_PUBLIC_URL="${LAB_PUBLIC_URL:-https://${DEFAULT_DOMAIN}/}"
+DEFAULT_GITEA_URL="${LAB_GITEA_URL:-https://${DEFAULT_DOMAIN}/git/jv/my-homelab-configs}"
+
+failures=0
+hosts_file=""
+
+cleanup() {
+    if [ -n "$hosts_file" ] && [ -f "$hosts_file" ]; then
+        rm -f "$hosts_file"
+    fi
+}
+trap cleanup EXIT
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+have() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+mark_fail() {
+    failures=$((failures + 1))
+    printf 'fail: %s\n' "$1"
+}
+
+inventory_scalar() {
+    local path="$1"
+
+    if have ruby && [ -f "$INVENTORY_FILE" ]; then
+        ruby -ryaml -e '
+          data = YAML.load_file(ARGV[0])
+          value = ARGV[1].split(".").reduce(data) { |memo, key| memo.respond_to?(:[]) ? memo[key] : nil }
+          puts value if value.is_a?(String)
+        ' "$INVENTORY_FILE" "$path" 2>/dev/null || true
+    fi
+}
+
+inventory_hosts() {
+    if have ruby && [ -f "$INVENTORY_FILE" ]; then
+        ruby -ryaml -e '
+          data = YAML.load_file(ARGV[0])
+          domain = data.fetch("domain", {})
+          hosts = [domain["base"]] + Array(domain["subdomains"])
+          puts hosts.compact.uniq
+        ' "$INVENTORY_FILE" 2>/dev/null || true
+    fi
+}
+
+domain_base="$(inventory_scalar domain.base)"
+public_url="$(inventory_scalar domain.public_url)"
+gitea_url="$(inventory_scalar domain.gitea_url)"
+
+domain_base="${domain_base:-$DEFAULT_DOMAIN}"
+public_url="${public_url:-$DEFAULT_PUBLIC_URL}"
+gitea_url="${gitea_url:-$DEFAULT_GITEA_URL}"
+
+hosts="$(inventory_hosts)"
+if [ -z "$hosts" ]; then
+    hosts="$domain_base"
+fi
+
+hosts_file="$(mktemp "${TMPDIR:-/tmp}/homelab-cert-hosts.XXXXXX")"
+printf '%s\n' "$hosts" >"$hosts_file"
+
+section "DNS Resolution"
+while IFS= read -r host; do
+    [ -n "$host" ] || continue
+    if have getent; then
+        if getent hosts "$host" >/dev/null 2>&1; then
+            printf '%-40s ok\n' "$host"
+        else
+            printf '%-40s fail\n' "$host"
+            failures=$((failures + 1))
+        fi
+    elif have dig; then
+        if dig "$host" A +time=3 +tries=1 +short | grep -q .; then
+            printf '%-40s ok\n' "$host"
+        else
+            printf '%-40s fail\n' "$host"
+            failures=$((failures + 1))
+        fi
+    else
+        echo "getent or dig is required for DNS checks."
+        failures=$((failures + 1))
+        break
+    fi
+done <"$hosts_file"
+
+section "TLS Expiry"
+while IFS= read -r host; do
+    [ -n "$host" ] || continue
+    if python3 - "$host" "$MIN_DAYS" <<'PY'
+import datetime
+import os
+import socket
+import ssl
+import sys
+
+host = sys.argv[1]
+minimum_days = int(sys.argv[2])
+context = ssl.create_default_context()
+
+try:
+    with socket.create_connection((host, 443), timeout=8) as sock:
+        with context.wrap_socket(sock, server_hostname=host) as tls_sock:
+            cert = tls_sock.getpeercert()
+except Exception as exc:
+    print(f"{host:<40} fail tls_connect={exc}")
+    sys.exit(1)
+
+not_after = datetime.datetime.strptime(
+    cert["notAfter"], "%b %d %H:%M:%S %Y %Z"
+).replace(tzinfo=datetime.timezone.utc)
+days = (not_after - datetime.datetime.now(datetime.timezone.utc)).days
+issuer_parts = cert.get("issuer", ())
+issuer = ",".join("=".join(item) for group in issuer_parts for item in group)
+status = "ok" if days >= minimum_days else "warn"
+print(f"{host:<40} {status} expires={not_after.date()} days={days} issuer={issuer}")
+sys.exit(0 if days >= minimum_days else 1)
+PY
+    then
+        :
+    else
+        failures=$((failures + 1))
+    fi
+done <"$hosts_file"
+
+section "Public HTTP"
+for url in "$public_url" "$gitea_url"; do
+    status="$(curl -k -sS -o /dev/null -w '%{http_code}' --max-time 10 "$url" 2>/dev/null || true)"
+    case "$status" in
+        200|301|302|307|308)
+            printf '%-60s ok http=%s\n' "$url" "$status"
+            ;;
+        *)
+            printf '%-60s fail http=%s\n' "$url" "${status:-none}"
+            failures=$((failures + 1))
+            ;;
+    esac
+done
+
+section "Edge IP Drift"
+edge_ip="$(inventory_scalar hosts.oci_edge.public_ip)"
+if [ -n "$edge_ip" ] && have dig; then
+    resolved="$(dig "$domain_base" A +time=3 +tries=1 +short | tail -1)"
+    if [ "$resolved" = "$edge_ip" ]; then
+        printf '%-40s ok resolved=%s\n' "$domain_base" "$resolved"
+    else
+        mark_fail "$domain_base resolves to ${resolved:-none}, expected OCI edge $edge_ip"
+    fi
+else
+    echo "Skipping edge IP drift check; dig or OCI edge inventory value is unavailable."
+fi
+
+if [ "$failures" -gt 0 ]; then
+    printf '\ncert-check found %s issue(s).\n' "$failures"
+    exit 1
+fi
+
+printf '\ncert-check found no blocking issues.\n'

scripts/change-journal

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+JOURNAL_DIR="${HOMELAB_CHANGE_JOURNAL_DIR:-${HOMELAB_STATE_DIR}/change-journal}"
+ACTION="${1:-list}"
+
+mkdir -p "$JOURNAL_DIR"
+
+timestamp() {
+    date -u +%Y%m%dT%H%M%SZ
+}
+
+latest_entries() {
+    local count="${1:-20}"
+
+    find "$JOURNAL_DIR" -maxdepth 1 -type f -name '*.log' -print 2>/dev/null |
+        sort |
+        tail -n "$count" |
+        while IFS= read -r file; do
+            printf '\n== %s ==\n' "$(basename "$file")"
+            sed -n '1,80p' "$file"
+        done
+}
+
+append_entry() {
+    local command_name="${2:-unknown}"
+    local detail="${3:-}"
+    local now
+    local file
+
+    now="$(timestamp)"
+    file="${JOURNAL_DIR}/${now}-${command_name//[^A-Za-z0-9_.-]/_}.log"
+
+    {
+        printf 'created_utc=%s\n' "$now"
+        printf 'command=%s\n' "$command_name"
+        printf 'detail=%s\n' "$detail"
+        printf 'host=%s\n' "$(hostname 2>/dev/null || echo unknown)"
+        printf 'repo=%s\n' "$REPO_ROOT"
+        printf 'branch=%s\n' "$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)"
+        printf 'revision=%s\n' "$(git -C "$REPO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown)"
+        printf 'dirty_files=%s\n' "$(git -C "$REPO_ROOT" status --short 2>/dev/null | wc -l | tr -d ' ')"
+        printf '\n-- git status --short --\n'
+        git -C "$REPO_ROOT" status --short 2>/dev/null || true
+    } >"$file"
+
+    printf 'Recorded change journal entry: %s\n' "$file"
+}
+
+case "$ACTION" in
+    append)
+        append_entry "$@"
+        ;;
+    list)
+        latest_entries "${2:-20}"
+        ;;
+    path)
+        printf '%s\n' "$JOURNAL_DIR"
+        ;;
+    *)
+        echo "Usage: ./jeannie change-journal {list [count]|path}" >&2
+        echo "Internal: scripts/change-journal append <command> [detail]" >&2
+        exit 1
+        ;;
+esac

scripts/control-plane

@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-/home/jv/.kube/config}}"
+CONTROL_PLANE_NODE="${LAB_CONTROL_PLANE_NODE:-debian}"
+TAINT_KEY="${LAB_CONTROL_PLANE_TAINT_KEY:-node-role.kubernetes.io/control-plane}"
+TAINT_EFFECT="${LAB_CONTROL_PLANE_TAINT_EFFECT:-NoSchedule}"
+
+usage() {
+    cat <<EOF
+Usage: ./jeannie control-plane {status|taint|untaint|pods}
+
+status   Show node taints and non-system pods currently scheduled on ${CONTROL_PLANE_NODE}.
+taint    Add ${TAINT_KEY}:${TAINT_EFFECT} to keep normal app/data-plane pods off the control plane.
+untaint  Remove ${TAINT_KEY}:${TAINT_EFFECT} from the control plane.
+pods     List non-system pods currently scheduled on ${CONTROL_PLANE_NODE}.
+
+Environment:
+  LAB_CONTROL_PLANE_NODE=${CONTROL_PLANE_NODE}
+EOF
+}
+
+require_kubectl() {
+    if ! command -v kubectl >/dev/null 2>&1; then
+        echo "kubectl is required." >&2
+        exit 1
+    fi
+    if [ ! -s "$KUBECONFIG_PATH" ]; then
+        echo "kubeconfig is missing: $KUBECONFIG_PATH" >&2
+        exit 1
+    fi
+    if ! kubectl --kubeconfig "$KUBECONFIG_PATH" get node "$CONTROL_PLANE_NODE" >/dev/null 2>&1; then
+        echo "control-plane node not found: ${CONTROL_PLANE_NODE}" >&2
+        echo "Set LAB_CONTROL_PLANE_NODE if the node has a different name." >&2
+        exit 1
+    fi
+}
+
+show_taints() {
+    kubectl --kubeconfig "$KUBECONFIG_PATH" get node "$CONTROL_PLANE_NODE" \
+        -o jsonpath='{range .spec.taints[*]}{.key}{"="}{.value}{":"}{.effect}{"\n"}{end}' |
+        sed '/^$/d' || true
+}
+
+show_non_system_pods() {
+    kubectl --kubeconfig "$KUBECONFIG_PATH" get pods -A \
+        --field-selector "spec.nodeName=${CONTROL_PLANE_NODE}" \
+        -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase' --no-headers |
+        awk '$1 !~ /^(kube-system|calico-system|tigera-operator)$/ { print }'
+}
+
+case "${1:-status}" in
+    status)
+        require_kubectl
+        echo "Node: ${CONTROL_PLANE_NODE}"
+        echo
+        echo "Taints:"
+        if ! show_taints | sed 's/^/  /'; then
+            true
+        fi
+        echo
+        echo "Non-system pods currently on control plane:"
+        if ! show_non_system_pods | sed 's/^/  /'; then
+            true
+        fi
+        ;;
+    taint)
+        require_kubectl
+        kubectl --kubeconfig "$KUBECONFIG_PATH" taint node "$CONTROL_PLANE_NODE" "${TAINT_KEY}:${TAINT_EFFECT}" --overwrite
+        echo
+        echo "Current taints:"
+        show_taints | sed 's/^/  /'
+        ;;
+    untaint)
+        require_kubectl
+        kubectl --kubeconfig "$KUBECONFIG_PATH" taint node "$CONTROL_PLANE_NODE" "${TAINT_KEY}:${TAINT_EFFECT}-" || true
+        echo
+        echo "Current taints:"
+        show_taints | sed 's/^/  /'
+        ;;
+    pods)
+        require_kubectl
+        show_non_system_pods
+        ;;
+    -h|--help|help)
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

scripts/gitops-status

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+NAMESPACE="${ARGOCD_NAMESPACE:-argocd}"
+LOG_PATTERN="${GITOPS_LOG_PATTERN:-error|failed|denied|timeout|authentication|unauthorized}"
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+have() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+kubectl_ok() {
+    have kubectl && kubectl version --client >/dev/null 2>&1
+}
+
+print_if_available() {
+    local description="$1"
+    shift
+
+    if "$@"; then
+        return 0
+    fi
+
+    printf '%s unavailable or returned no data.\n' "$description"
+}
+
+if ! kubectl_ok; then
+    echo "kubectl is required for gitops-status." >&2
+    exit 1
+fi
+
+if ! kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
+    echo "Argo CD namespace '$NAMESPACE' was not found or the Kubernetes API is unreachable." >&2
+    exit 1
+fi
+
+section "Argo CD Applications"
+if ! kubectl -n "$NAMESPACE" get applications.argoproj.io \
+    -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REVISION:.status.sync.revision,PATH:.spec.source.path' \
+    --no-headers 2>/dev/null; then
+    echo "No Argo CD Applications were found."
+fi
+
+section "Out Of Sync Or Degraded"
+apps_report="$(
+    kubectl -n "$NAMESPACE" get applications.argoproj.io \
+        -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,MESSAGE:.status.conditions[-1].message' \
+        --no-headers 2>/dev/null || true
+)"
+
+if [ -z "$apps_report" ]; then
+    echo "No application status data available."
+else
+    printf '%s\n' "$apps_report" | awk '$2 != "Synced" || $3 != "Healthy" { print }'
+fi
+
+section "Repository Secrets"
+print_if_available "Repository secrets" \
+    kubectl -n "$NAMESPACE" get secrets -l argocd.argoproj.io/secret-type=repository \
+    -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.argocd\.argoproj\.io/secret-type' \
+    --no-headers
+
+section "Recent Argo CD Warnings"
+print_if_available "Argo CD events" \
+    sh -c "kubectl -n '$NAMESPACE' get events --sort-by=.lastTimestamp 2>/dev/null | tail -40"
+
+section "Recent Repo Server Errors"
+if kubectl -n "$NAMESPACE" get deployment argocd-repo-server >/dev/null 2>&1; then
+    kubectl -n "$NAMESPACE" logs deployment/argocd-repo-server --tail=120 2>/dev/null |
+        grep -Ei "$LOG_PATTERN" || echo "No recent repo-server errors matched '$LOG_PATTERN'."
+else
+    echo "argocd-repo-server deployment not found."
+fi
+
+section "Recent Application Controller Errors"
+if kubectl -n "$NAMESPACE" get statefulset argocd-application-controller >/dev/null 2>&1; then
+    kubectl -n "$NAMESPACE" logs statefulset/argocd-application-controller --tail=120 2>/dev/null |
+        grep -Ei "$LOG_PATTERN" || echo "No recent application-controller errors matched '$LOG_PATTERN'."
+else
+    echo "argocd-application-controller statefulset not found."
+fi

scripts/golden-ledger

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+LEDGER="${HOMELAB_GOLDEN_LEDGER:-${REPO_ROOT}/infra/pimox/golden-image-ledger.yml}"
+
+case "${1:-show}" in
+    show)
+        sed -n '1,220p' "$LEDGER"
+        ;;
+    check)
+        if [ ! -s "$LEDGER" ]; then
+            echo "missing ledger: $LEDGER" >&2
+            exit 1
+        fi
+        python3 - "$LEDGER" <<'PY'
+import sys
+
+try:
+    import yaml
+except ImportError:
+    print("yaml module unavailable; ledger file exists")
+    raise SystemExit(0)
+
+path = sys.argv[1]
+with open(path, encoding="utf-8") as handle:
+    doc = yaml.safe_load(handle) or {}
+
+required = [
+    "template.vmid",
+    "template.name",
+    "os.distribution",
+    "os.release",
+    "os.architecture",
+    "kubernetes.kubelet",
+    "kubernetes.kubeadm",
+    "kubernetes.kubectl",
+]
+missing = []
+for dotted in required:
+    value = doc
+    for part in dotted.split("."):
+        value = value.get(part) if isinstance(value, dict) else None
+    if value in (None, ""):
+        missing.append(dotted)
+
+if missing:
+    print("missing required ledger fields:")
+    for item in missing:
+        print(f"  {item}")
+    raise SystemExit(1)
+
+print(f"ledger ok: {path}")
+PY
+        ;;
+    update-stamp)
+        echo "Manual edit is preferred so image URLs/checksums stay reviewable: $LEDGER" >&2
+        exit 1
+        ;;
+    *)
+        echo "Usage: ./jeannie golden-ledger {show|check}" >&2
+        exit 1
+        ;;
+esac

scripts/grafana-dashboards

@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+DASHBOARD_DIR="${HOMELAB_GRAFANA_DASHBOARD_DIR:-${REPO_ROOT}/bootstrap/platform/grafana-dashboards}"
+NAMESPACE="${HOMELAB_GRAFANA_NAMESPACE:-monitoring}"
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-${HOME}/.kube/config}}"
+
+usage() {
+    cat <<EOF
+Usage: ./jeannie grafana-dashboards {list|apply}
+
+list   Show repo-managed dashboard files.
+apply  Rebuild Grafana dashboard ConfigMaps and restart Grafana.
+EOF
+}
+
+list_dashboards() {
+    find "$DASHBOARD_DIR" -maxdepth 1 -type f -name '*.json' -print | sort
+}
+
+apply_dashboards() {
+    local file
+    local base
+    local name
+    local grafana_workload=""
+
+    if ! command -v kubectl >/dev/null 2>&1; then
+        echo "kubectl is required." >&2
+        exit 1
+    fi
+    if [ ! -s "$KUBECONFIG_PATH" ]; then
+        echo "kubeconfig is missing: $KUBECONFIG_PATH" >&2
+        exit 1
+    fi
+
+    while IFS= read -r file; do
+        base="$(basename "$file")"
+        name="homelab-grafana-${base%.json}"
+        echo "Applying Grafana dashboard ConfigMap ${NAMESPACE}/${name} from ${base}..."
+        kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" create configmap "$name" \
+            --from-file="$base=$file" \
+            --dry-run=client -o yaml |
+            kubectl --kubeconfig "$KUBECONFIG_PATH" apply -f -
+        kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" label configmap "$name" grafana_dashboard=1 --overwrite
+        kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" annotate configmap "$name" grafana_folder=Homelab --overwrite
+    done < <(list_dashboards)
+
+    echo "Restarting Grafana so the sidecar reloads dashboards..."
+    if kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" get deployment prometheus-stack-grafana >/dev/null 2>&1; then
+        grafana_workload="deployment/prometheus-stack-grafana"
+    elif kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" get statefulset prometheus-stack-grafana >/dev/null 2>&1; then
+        grafana_workload="statefulset/prometheus-stack-grafana"
+    else
+        grafana_workload="$(
+            kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" get deployment,statefulset \
+                -l app.kubernetes.io/name=grafana \
+                -o name 2>/dev/null | head -n 1
+        )"
+    fi
+
+    if [ -z "$grafana_workload" ]; then
+        echo "Grafana workload was not found in namespace ${NAMESPACE}; dashboards were applied but Grafana was not restarted." >&2
+        echo "Check with: kubectl -n ${NAMESPACE} get deploy,sts | grep grafana" >&2
+        return 1
+    fi
+
+    kubectl --kubeconfig "$KUBECONFIG_PATH" -n "$NAMESPACE" rollout restart "$grafana_workload"
+}
+
+case "${1:-list}" in
+    list)
+        list_dashboards
+        ;;
+    apply|rebuild)
+        apply_dashboards
+        ;;
+    -h|--help|help)
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

scripts/homelab-map

@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+INVENTORY_FILE="${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}"
+FORMAT="${1:-text}"
+
+inventory() {
+    local path="$1"
+    local fallback="$2"
+
+    if command -v ruby >/dev/null 2>&1 && [ -f "$INVENTORY_FILE" ]; then
+        value="$(
+            ruby -ryaml -e '
+              data = YAML.load_file(ARGV[0])
+              value = ARGV[1].split(".").reduce(data) { |memo, key| memo.respond_to?(:[]) ? memo[key] : nil }
+              puts value if value.is_a?(String) || value.is_a?(Integer)
+            ' "$INVENTORY_FILE" "$path" 2>/dev/null || true
+        )"
+        if [ -n "$value" ]; then
+            printf '%s\n' "$value"
+            return 0
+        fi
+    fi
+
+    printf '%s\n' "$fallback"
+}
+
+domain="$(inventory domain.base lab2025.duckdns.org)"
+debian="$(inventory hosts.debian.lan_ip 192.168.100.73)"
+debian_ts="$(inventory hosts.debian.tailscale_ip 100.85.138.30)"
+rpi="$(inventory hosts.rpi4.lan_ip 192.168.100.89)"
+oci="$(inventory hosts.oci_edge.public_ip 132.145.170.74)"
+oci_ts="$(inventory hosts.oci_edge.tailscale_ip 100.118.255.19)"
+pimox="$(inventory hosts.opi5_pimox.lan_ip 192.168.100.80)"
+traefik="$(inventory network.metallb.traefik_ip 192.168.100.240)"
+registry="$(inventory services.local_registry.endpoint 192.168.100.73:30500)"
+gitea_http="$(inventory services.gitea.http_port 3000)"
+gitea_ssh="$(inventory services.gitea.ssh_port 32222)"
+pihole_web="$(inventory services.rpi_dns.pihole_web_port 8081)"
+kuma="$(inventory services.rpi_dns.uptime_kuma_port 3001)"
+
+if [ "$FORMAT" = "--dot" ] || [ "$FORMAT" = "dot" ]; then
+    cat <<EOF
+digraph homelab {
+  rankdir=LR;
+  "Internet" -> "OCI edge ${oci}";
+  "OCI edge ${oci}" -> "Traefik ${traefik}";
+  "OCI edge ${oci}" -> "Debian Tailscale ${debian_ts}";
+  "Debian ${debian}" -> "Gitea :${gitea_http}/git";
+  "Debian ${debian}" -> "Local registry ${registry}";
+  "Debian ${debian}" -> "Kubernetes control plane";
+  "Debian ${debian}" -> "Pimox ${pimox}";
+  "Pimox ${pimox}" -> "Worker VMs";
+  "Worker VMs" -> "Kubernetes apps";
+  "Kubernetes apps" -> "Traefik ${traefik}";
+  "Gitea :${gitea_http}/git" -> "Argo CD";
+  "Argo CD" -> "Kubernetes apps";
+  "Clients" -> "Pi-hole ${rpi}:53";
+  "Pi-hole ${rpi}:53" -> "Unbound";
+  "Pi-hole ${rpi}:53" -> "Public DNS fallback";
+  "Uptime Kuma ${rpi}:${kuma}" -> "Public URLs";
+  "Uptime Kuma ${rpi}:${kuma}" -> "LAN services";
+}
+EOF
+    exit 0
+fi
+
+cat <<EOF
+Homelab dependency map
+======================
+
+Public entry
+  Internet
+    -> ${domain}
+    -> OCI edge ${oci} / tailscale ${oci_ts}
+    -> Traefik LoadBalancer ${traefik}
+    -> Kubernetes ingress routes
+    -> website, demos, Heimdall, observability, Argo CD, arr routes
+
+Git and automation
+  Debian ${debian}
+    -> Gitea HTTP :${gitea_http}/git and SSH :${gitea_ssh}
+    -> Gitea Actions runner
+    -> local registry ${registry}
+    -> jeannie orchestration
+  Gitea repository
+    -> Argo CD applications
+    -> Kubernetes manifests under apps/
+    -> edge/provisioning/cluster/platform Terraform stacks
+
+Cluster foundation
+  Debian ${debian}
+    -> kubeadm control plane
+    -> container registry
+    -> provisioning HTTP/TFTP
+  Orange Pi Pimox ${pimox}
+    -> worker VM template
+    -> worker VM clones
+    -> Kubernetes worker capacity
+
+DNS and always-on health
+  Clients and lab hosts
+    -> RPi4 Pi-hole ${rpi}:53
+    -> Unbound resolver
+    -> public fallback DNS if Unbound is unhealthy
+  RPi4
+    -> Pi-hole web ${rpi}:${pihole_web}
+    -> Uptime Kuma ${rpi}:${kuma}
+    -> monitors public URLs, LAN services, DNS, Traefik, Gitea, arr
+
+Observability and operations
+  Prometheus stack
+    -> nodes, pods, Traefik, website probes, Gitea/Pi-hole scrape coverage
+    -> homelab-alerts
+    -> Grafana dashboards
+  jeannie status
+    -> host/bootstrap checks
+    -> local services
+    -> RPi DNS and Kuma
+    -> Pimox and Kubernetes
+    -> platform/apps
+    -> edge/public URLs
+
+Security learning loop
+  security-lab namespace
+    -> isolated OWASP practice targets
+    -> ZAP/nuclei/trivy/lynis helper commands
+    -> report-only findings before manual fixes
+
+Use './jeannie map --dot' to emit a Graphviz DOT version.
+EOF

scripts/kubeconfig-readonly

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ADMIN_KUBECONFIG="${KUBECONFIG:-${KUBECONFIG_PATH:-${HOME}/.kube/config}}"
+READONLY_NAMESPACE="${READONLY_NAMESPACE:-homelab-access}"
+READONLY_SERVICE_ACCOUNT="${READONLY_SERVICE_ACCOUNT:-homelab-readonly}"
+READONLY_KUBECONFIG="${READONLY_KUBECONFIG:-${HOME}/.kube/homelab-readonly.conf}"
+READONLY_TOKEN_DURATION="${READONLY_TOKEN_DURATION:-24h}"
+
+usage() {
+    cat <<'EOF'
+Usage: scripts/kubeconfig-readonly
+
+Generate a separate read-only kubeconfig from the repo-managed
+homelab-access/homelab-readonly ServiceAccount.
+EOF
+}
+
+require_kubectl() {
+    if ! command -v kubectl >/dev/null 2>&1; then
+        echo "kubectl is required." >&2
+        exit 1
+    fi
+    if [[ ! -s "${ADMIN_KUBECONFIG}" ]]; then
+        echo "admin kubeconfig missing: ${ADMIN_KUBECONFIG}" >&2
+        exit 1
+    fi
+}
+
+main() {
+    local server
+    local cluster_name
+    local ca_data
+    local token
+    local user_name="homelab-readonly"
+    local context_name="homelab-readonly"
+
+    case "${1:-}" in
+        -h | --help | help)
+            usage
+            return 0
+            ;;
+    esac
+
+    require_kubectl
+
+    kubectl --kubeconfig "${ADMIN_KUBECONFIG}" -n "${READONLY_NAMESPACE}" get serviceaccount "${READONLY_SERVICE_ACCOUNT}" >/dev/null
+    token="$(kubectl --kubeconfig "${ADMIN_KUBECONFIG}" -n "${READONLY_NAMESPACE}" create token "${READONLY_SERVICE_ACCOUNT}" --duration="${READONLY_TOKEN_DURATION}")"
+    server="$(kubectl --kubeconfig "${ADMIN_KUBECONFIG}" config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
+    cluster_name="$(kubectl --kubeconfig "${ADMIN_KUBECONFIG}" config view --minify -o jsonpath='{.clusters[0].name}')"
+    ca_data="$(kubectl --kubeconfig "${ADMIN_KUBECONFIG}" config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')"
+
+    mkdir -p "$(dirname "${READONLY_KUBECONFIG}")"
+    umask 077
+    cat >"${READONLY_KUBECONFIG}" <<EOF
+apiVersion: v1
+kind: Config
+clusters:
+  - name: ${cluster_name}
+    cluster:
+      server: ${server}
+      certificate-authority-data: ${ca_data}
+users:
+  - name: ${user_name}
+    user:
+      token: ${token}
+contexts:
+  - name: ${context_name}
+    context:
+      cluster: ${cluster_name}
+      user: ${user_name}
+      namespace: default
+current-context: ${context_name}
+EOF
+    chmod 0600 "${READONLY_KUBECONFIG}"
+
+    echo "Created read-only kubeconfig: ${READONLY_KUBECONFIG}"
+    echo "Test with:"
+    echo "  KUBECONFIG=${READONLY_KUBECONFIG} kubectl get pods -A"
+}
+
+main "$@"

scripts/recover-plan

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+KUBECONFIG="${KUBECONFIG:-${KUBECONFIG_PATH:-${HOME}/.kube/config}}"
+
+check() {
+    local label="$1"
+    shift
+
+    printf '%-36s ' "${label}"
+    if "$@" >/tmp/homelab-recover-check.$$ 2>&1; then
+        printf 'ok\n'
+    else
+        printf 'check\n'
+        sed -n '1,3p' /tmp/homelab-recover-check.$$ | sed 's/^/    /'
+    fi
+    rm -f /tmp/homelab-recover-check.$$
+}
+
+has_gitea_backup() {
+    find "${LAB_GITEA_BACKUP_DIR:-/home/jv/backups/gitea}" -maxdepth 1 -type f -name 'gitea-*.zip' -print -quit | grep -q .
+}
+
+has_state_backup() {
+    find "${HOMELAB_TOFU_STATE_BACKUP_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab/tofu-state-backups}" -maxdepth 1 -type f -name 'tofu-state-*.tgz' -print -quit | grep -q .
+}
+
+main() {
+    cat <<'EOF'
+== Homelab Recovery Order ==
+
+1. Restore Debian control host baseline.
+   Command: ./jeannie validate
+   Check: Docker, containerd, Tailscale, kubeconfig path, repo checkout.
+
+2. Restore or verify Gitea.
+   Commands:
+     ./jeannie deploy-gitea
+     ./jeannie drill-gitea-restore
+     ./jeannie bootstrap-gitea-repo
+
+3. Restore RPi DNS services.
+   Commands:
+     ./jeannie rpi-services
+     ./jeannie doctor-rpi
+     ./jeannie drill-pihole-restore
+
+4. Verify Pimox template and worker storage.
+   Commands:
+     ./jeannie preflight
+     ./jeannie doctor-preapply
+
+5. Rebuild or start Kubernetes.
+   If state exists and nodes are stopped: ./jeannie start-cluster
+   If cluster must be rebuilt:          ./jeannie rebuild-cluster
+
+6. Restore platform and apps from GitOps.
+   Commands:
+     ./jeannie apps
+     ./jeannie gitops-status
+
+7. Verify public edge.
+   Commands:
+     ./jeannie doctor-edge
+     ./jeannie cert-check
+     ./jeannie status
+
+8. Capture final recovery evidence.
+   Commands:
+     ./jeannie release-snapshot
+     ./jeannie backup-gitea
+     ./jeannie state-backup
+
+EOF
+
+    echo "== Current Recovery Inputs =="
+    check "repo validation" "${REPO_ROOT}/scripts/validate-homelab"
+    check "Gitea backup archive" has_gitea_backup
+    check "OpenTofu state backup" has_state_backup
+    check "Pi-hole restore inputs" "${REPO_ROOT}/scripts/restore-drill" pihole
+    check "Kubernetes API" kubectl --kubeconfig "${KUBECONFIG}" get --raw=/readyz
+
+    cat <<'EOF'
+
+== Recovery Rule ==
+Prefer restoring source-of-truth first: repo, Gitea, DNS, state backups, then
+rebuild compute. Treat Pimox worker disks as rebuildable unless a service has an
+explicit backup and restore path.
+EOF
+}
+
+main "$@"

scripts/recover-power

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+DRY_RUN=false
+
+if [ "${1:-}" = "--dry-run" ]; then
+    DRY_RUN=true
+fi
+
+run_step() {
+    local label="$1"
+    shift
+
+    printf '\n== %s ==\n' "$label"
+    printf '+ %s\n' "$*"
+    if [ "$DRY_RUN" = "true" ]; then
+        return 0
+    fi
+    "$@"
+}
+
+run_optional() {
+    local label="$1"
+    shift
+
+    if ! run_step "$label" "$@"; then
+        printf 'warn: step failed: %s\n' "$label" >&2
+    fi
+}
+
+cat <<EOF
+Power-loss recovery order
+=========================
+
+This sequence restores dependencies before dependents:
+Debian runtime -> Gitea -> RPi DNS -> Pimox workers -> Kubernetes -> GitOps/apps -> edge/public URLs.
+EOF
+
+run_optional "Debian runtime status" systemctl is-active docker containerd
+run_optional "Start Docker runtime" sudo systemctl start docker containerd
+run_optional "Deploy or verify Gitea" "${REPO_ROOT}/jeannie" deploy-gitea
+run_optional "Verify RPi services" "${REPO_ROOT}/jeannie" doctor-rpi
+run_optional "Start Kubernetes cluster" "${REPO_ROOT}/jeannie" start-cluster
+run_optional "Cluster doctor" "${REPO_ROOT}/jeannie" doctor-cluster
+run_optional "GitOps status" "${REPO_ROOT}/jeannie" gitops-status
+run_optional "Public edge doctor" "${REPO_ROOT}/jeannie" doctor-edge
+run_optional "Final status cascade" "${REPO_ROOT}/jeannie" status
+
+printf '\nRecovery sequence complete%s.\n' "$([ "$DRY_RUN" = "true" ] && printf ' dry-run' || true)"

scripts/release-snapshot

@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+SNAPSHOT_DIR="${HOMELAB_RELEASE_SNAPSHOT_DIR:-${HOMELAB_STATE_DIR}/release-snapshots}"
+TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)"
+REPORT="${SNAPSHOT_DIR}/release-snapshot-${TIMESTAMP}.txt"
+
+mkdir -p "$SNAPSHOT_DIR"
+exec > >(tee "$REPORT") 2>&1
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+have() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+run_optional() {
+    local description="$1"
+    shift
+
+    section "$description"
+    if "$@"; then
+        return 0
+    fi
+
+    printf 'warn: %s unavailable or failed\n' "$description"
+}
+
+latest_file() {
+    local dir="$1"
+    local pattern="$2"
+
+    if [ -d "$dir" ]; then
+        find "$dir" -maxdepth 1 -type f -name "$pattern" -print 2>/dev/null | sort | tail -1
+    fi
+}
+
+section "Snapshot Metadata"
+printf 'created_utc=%s\n' "$TIMESTAMP"
+printf 'report=%s\n' "$REPORT"
+printf 'repo=%s\n' "$REPO_ROOT"
+printf 'host=%s\n' "$(hostname 2>/dev/null || echo unknown)"
+
+section "Git State"
+git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD
+git -C "$REPO_ROOT" rev-parse HEAD
+git -C "$REPO_ROOT" status --short
+git -C "$REPO_ROOT" log --oneline -5
+
+if have kubectl; then
+    run_optional "Kubernetes Nodes" kubectl get nodes -o wide
+    run_optional "Kubernetes Problem Pods" sh -c \
+        'kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide'
+    run_optional "Argo CD Applications" sh -c \
+        'kubectl -n argocd get applications.argoproj.io -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status,REVISION:.status.sync.revision,PATH:.spec.source.path --no-headers'
+    run_optional "Workload Images" sh -c \
+        'kubectl get deploy,statefulset,daemonset -A -o jsonpath="{range .items[*]}{.metadata.namespace}{\"/\"}{.metadata.name}{\"\\t\"}{range .spec.template.spec.containers[*]}{.image}{\" \"}{end}{\"\\n\"}{end}"'
+else
+    section "Kubernetes"
+    echo "kubectl not installed; skipping Kubernetes snapshot."
+fi
+
+if have helm; then
+    run_optional "Helm Releases" helm list -A
+else
+    section "Helm Releases"
+    echo "helm not installed; skipping Helm release snapshot."
+fi
+
+section "Backup Pointers"
+state_backup="$(latest_file "${HOMELAB_TOFU_STATE_BACKUP_DIR:-${HOMELAB_STATE_DIR}/tofu-state-backups}" 'tofu-state-*.tgz' || true)"
+gitea_backup="$(latest_file "${LAB_GITEA_BACKUP_DIR:-/home/jv/backups/gitea}" 'gitea-*.zip' || true)"
+printf 'latest_tofu_state_backup=%s\n' "${state_backup:-missing}"
+printf 'latest_gitea_backup=%s\n' "${gitea_backup:-missing}"
+
+section "Public URLs"
+for url in "${LAB_PUBLIC_URL:-https://lab2025.duckdns.org/}" "${LAB_GITEA_URL:-https://lab2025.duckdns.org/git/jv/my-homelab-configs}"; do
+    if have curl; then
+        status="$(curl -k -sS -o /dev/null -w '%{http_code}' --max-time 10 "$url" 2>/dev/null || true)"
+        printf '%-60s http=%s\n' "$url" "${status:-none}"
+    else
+        printf '%-60s curl not installed\n' "$url"
+    fi
+done
+
+section "Next Actions"
+cat <<EOF
+Before applying risky changes:
+1. Review this report for dirty Git state, unhealthy apps, or missing backups.
+2. Run ./jeannie state-backup if latest_tofu_state_backup is missing or old.
+3. Run ./jeannie backup-gitea if latest_gitea_backup is missing or old.
+4. Keep this report path with the change notes: $REPORT
+EOF

scripts/report-render

@@ -0,0 +1,156 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODE="compact"
+ONLY="all"
+TITLE="${REPORT_TITLE:-Jeannie Report}"
+INPUT_FILE=""
+
+cleanup() {
+    if [ -n "$INPUT_FILE" ] && [ -f "$INPUT_FILE" ]; then
+        rm -f "$INPUT_FILE"
+    fi
+}
+trap cleanup EXIT
+
+usage() {
+    cat <<EOF
+Usage: report-render [--title TITLE] [--details] [--only failures|warnings|problems|all] [--json]
+
+Reads TSV rows from stdin:
+  status<TAB>area<TAB>check<TAB>summary<TAB>detail<TAB>fix<TAB>graph<TAB>query
+
+Statuses: ok, warn, fail, skip
+EOF
+}
+
+while (($# > 0)); do
+    case "$1" in
+        --title)
+            TITLE="${2:-}"
+            shift 2
+            ;;
+        --details)
+            MODE="details"
+            shift
+            ;;
+        --only)
+            ONLY="${2:-all}"
+            shift 2
+            ;;
+        --json)
+            MODE="json"
+            shift
+            ;;
+        -h|--help)
+            usage
+            exit 0
+            ;;
+        *)
+            usage >&2
+            exit 1
+            ;;
+    esac
+done
+
+INPUT_FILE="$(mktemp "${TMPDIR:-/tmp}/jeannie-report.XXXXXX")"
+cat >"$INPUT_FILE"
+
+python3 - "$TITLE" "$MODE" "$ONLY" "$INPUT_FILE" <<'PY'
+import json
+import sys
+from collections import OrderedDict
+
+title, mode, only, input_file = sys.argv[1:5]
+rows = []
+
+with open(input_file, encoding="utf-8") as handle:
+    for raw in handle:
+        raw = raw.rstrip("\n")
+        if not raw:
+            continue
+        parts = raw.split("\t")
+        while len(parts) < 8:
+            parts.append("")
+        status, area, check, summary, detail, fix, graph, query = parts[:8]
+        status = status.strip().lower() or "skip"
+        if status == "ok":
+            graph = ""
+            query = ""
+        rows.append(
+            {
+                "status": status,
+                "area": area.strip() or "General",
+                "check": check.strip(),
+                "summary": summary.strip(),
+                "detail": detail.strip(),
+                "fix": fix.strip(),
+                "graph": graph.strip(),
+                "query": query.strip(),
+            }
+        )
+
+def include(row):
+    status = row["status"]
+    if only == "all":
+        return True
+    if only == "failures":
+        return status == "fail"
+    if only == "warnings":
+        return status == "warn"
+    if only == "problems":
+        return status in {"fail", "warn"}
+    return True
+
+rows = [row for row in rows if include(row)]
+
+if mode == "json":
+    print(json.dumps({"title": title, "rows": rows}, indent=2))
+    raise SystemExit(0)
+
+counts = {"fail": 0, "warn": 0, "ok": 0, "skip": 0}
+for row in rows:
+    counts[row["status"]] = counts.get(row["status"], 0) + 1
+
+print(title)
+print("=" * len(title))
+print(
+    f"fail={counts.get('fail', 0)}  "
+    f"warn={counts.get('warn', 0)}  "
+    f"ok={counts.get('ok', 0)}  "
+    f"skip={counts.get('skip', 0)}"
+)
+
+groups = OrderedDict()
+for row in rows:
+    groups.setdefault(row["area"], []).append(row)
+
+status_rank = {"fail": 0, "warn": 1, "skip": 2, "ok": 3}
+status_label = {"fail": "fail", "warn": "warn", "ok": "ok", "skip": "skip"}
+
+for area, area_rows in groups.items():
+    print(f"\n== {area} ==")
+    for row in sorted(area_rows, key=lambda item: (status_rank.get(item["status"], 9), item["check"])):
+        status = status_label.get(row["status"], row["status"])
+        check = row["check"][:30]
+        summary = row["summary"]
+        print(f"{status:<5} {check:<30} {summary}")
+        if mode == "details" or row["status"] in {"fail", "warn"}:
+            if row["detail"]:
+                print(f"      detail: {row['detail']}")
+            if row["fix"]:
+                print(f"      fix:    {row['fix']}")
+            if row["graph"]:
+                print(f"      graph:  {row['graph']}")
+            if row["query"]:
+                print(f"      query:  {row['query']}")
+
+problems = [row for row in rows if row["status"] in {"fail", "warn"}]
+if problems:
+    first = problems[0]
+    print("\nNext Fix")
+    if first["fix"]:
+        print(first["fix"])
+    else:
+        print(f"Investigate {first['area']} / {first['check']}")
+PY

scripts/resource-budget

@@ -0,0 +1,247 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+BUDGET_FILE="${HOMELAB_RESOURCE_BUDGET_FILE:-${REPO_ROOT}/infra/resource-budgets.yml}"
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-/home/jv/.kube/config}}"
+DETAILS=false
+MAX_EXAMPLES="${LAB_RESOURCE_BUDGET_EXAMPLES:-12}"
+
+failures=0
+warnings=0
+
+usage() {
+    cat <<'EOF'
+Usage: ./jeannie resource-budget [--details]
+
+Print host and Kubernetes resource budget checks.
+
+Options:
+  --details  Show every pod/container missing requests or limits.
+EOF
+}
+
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+        --details|--verbose)
+            DETAILS=true
+            ;;
+        -h|--help|help)
+            usage
+            exit 0
+            ;;
+        *)
+            usage >&2
+            exit 2
+            ;;
+    esac
+    shift
+done
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+warn() {
+    warnings=$((warnings + 1))
+    printf 'warn - %s\n' "$1"
+}
+
+fail() {
+    failures=$((failures + 1))
+    printf 'fail - %s\n' "$1"
+}
+
+have() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+host_budget() {
+    section "Host Budget"
+
+    if have free; then
+        free -m | awk '/^Mem:/ {
+          used_pct = int(($3 / $2) * 100)
+          printf "Debian memory used        %s%%\n", used_pct
+          if (used_pct > 80) exit 2
+        }' || warn "Debian memory usage is above the default 80% budget"
+    else
+        warn "free command unavailable"
+    fi
+
+    if have df; then
+        df -P / /data 2>/dev/null | awk 'NR > 1 { gsub("%", "", $5); free = 100 - $5; printf "%-25s free=%s%%\n", $6, free }' || true
+    else
+        warn "df command unavailable"
+    fi
+}
+
+kubernetes_budget() {
+    section "Kubernetes Budget"
+
+    if ! have kubectl || [ ! -s "$KUBECONFIG_PATH" ]; then
+        warn "kubectl or kubeconfig unavailable; skipping Kubernetes budget checks"
+        return 0
+    fi
+    if ! kubectl --kubeconfig "$KUBECONFIG_PATH" get --raw=/readyz >/dev/null 2>&1; then
+        warn "Kubernetes API unavailable; skipping Kubernetes budget checks"
+        return 0
+    fi
+
+    python3 - "$BUDGET_FILE" "$KUBECONFIG_PATH" "$DETAILS" "$MAX_EXAMPLES" <<'PY'
+import json
+import subprocess
+import sys
+from collections import Counter, defaultdict
+
+budget_file, kubeconfig, details_arg, max_examples_arg = sys.argv[1:5]
+show_details = details_arg.lower() == "true"
+max_examples = int(max_examples_arg)
+
+try:
+    import yaml
+except ImportError:
+    yaml = None
+
+def load_budget(path):
+    if yaml:
+        with open(path, encoding="utf-8") as handle:
+            return yaml.safe_load(handle) or {}
+    return {
+        "cluster": {
+            "max_total_cpu_millicores": 9000,
+            "max_total_memory_mib": 22000,
+            "require_requests": True,
+            "require_limits": True,
+        },
+        "namespaces": {},
+    }
+
+def cpu_to_m(value):
+    if not value:
+        return 0
+    text = str(value)
+    if text.endswith("m"):
+        return int(text[:-1])
+    return int(float(text) * 1000)
+
+def mem_to_mib(value):
+    if not value:
+        return 0
+    text = str(value)
+    units = {
+        "Ki": 1 / 1024,
+        "Mi": 1,
+        "Gi": 1024,
+        "Ti": 1024 * 1024,
+        "K": 1 / 1000,
+        "M": 1000 / 1024,
+        "G": 1000 * 1000 / 1024,
+    }
+    for suffix, multiplier in units.items():
+        if text.endswith(suffix):
+            return int(float(text[: -len(suffix)]) * multiplier)
+    return int(float(text) / (1024 * 1024))
+
+budget = load_budget(budget_file)
+cluster_budget = budget.get("cluster", {})
+namespace_budgets = budget.get("namespaces", {})
+raw = subprocess.check_output(
+    ["kubectl", "--kubeconfig", kubeconfig, "get", "pods", "-A", "-o", "json"],
+    text=True,
+)
+pods = json.loads(raw).get("items", [])
+namespace_totals = {}
+missing = []
+missing_by_namespace = defaultdict(Counter)
+
+for pod in pods:
+    ns = pod["metadata"]["namespace"]
+    pod_name = pod["metadata"]["name"]
+    namespace_totals.setdefault(ns, {"cpu": 0, "memory": 0})
+    for container in pod.get("spec", {}).get("containers", []):
+        name = container["name"]
+        resources = container.get("resources", {})
+        requests = resources.get("requests", {})
+        limits = resources.get("limits", {})
+        namespace_totals[ns]["cpu"] += cpu_to_m(requests.get("cpu"))
+        namespace_totals[ns]["memory"] += mem_to_mib(requests.get("memory"))
+        if cluster_budget.get("require_requests", True) and ("cpu" not in requests or "memory" not in requests):
+            missing.append(f"{ns}/{pod_name}:{name} missing requests")
+            missing_by_namespace[ns]["requests"] += 1
+        if cluster_budget.get("require_limits", True) and ("cpu" not in limits or "memory" not in limits):
+            missing.append(f"{ns}/{pod_name}:{name} missing limits")
+            missing_by_namespace[ns]["limits"] += 1
+
+total_cpu = sum(item["cpu"] for item in namespace_totals.values())
+total_memory = sum(item["memory"] for item in namespace_totals.values())
+print(f"cluster_requests_cpu_m={total_cpu}")
+print(f"cluster_requests_memory_mib={total_memory}")
+
+exit_code = 0
+if total_cpu > int(cluster_budget.get("max_total_cpu_millicores", 9000)):
+    print("fail - cluster CPU requests exceed budget")
+    exit_code = 1
+if total_memory > int(cluster_budget.get("max_total_memory_mib", 22000)):
+    print("fail - cluster memory requests exceed budget")
+    exit_code = 1
+
+for ns, totals in sorted(namespace_totals.items()):
+    ns_budget = namespace_budgets.get(ns)
+    if not ns_budget:
+        continue
+    print(f"{ns}: cpu_m={totals['cpu']} memory_mib={totals['memory']}")
+    if totals["cpu"] > int(ns_budget.get("max_cpu_millicores", 999999)):
+        print(f"fail - {ns} CPU requests exceed budget")
+        exit_code = 1
+    if totals["memory"] > int(ns_budget.get("max_memory_mib", 999999)):
+        print(f"fail - {ns} memory requests exceed budget")
+        exit_code = 1
+
+if missing:
+    print(f"missing_resource_policy: total={len(missing)}")
+    print("by_namespace:")
+    for ns, counts in sorted(
+        missing_by_namespace.items(),
+        key=lambda item: (item[1]["requests"] + item[1]["limits"], item[0]),
+        reverse=True,
+    ):
+        print(f"  {ns}: requests_missing={counts['requests']} limits_missing={counts['limits']}")
+
+    if show_details:
+        print("details:")
+        for item in missing:
+            print(f"  {item}")
+    else:
+        print("examples:")
+        for item in missing[:max_examples]:
+            print(f"  {item}")
+        if len(missing) > max_examples:
+            print(f"  ... {len(missing) - max_examples} more")
+            print("  run ./jeannie resource-budget --details for the full list")
+        print("next_steps:")
+        print("  1. Add requests before limits for app workloads you own.")
+        print("  2. Leave kube-system/control-plane static pods for a separate pass.")
+        print("  3. Use ./jeannie control-plane taint after Pimox workers are Ready.")
+    exit_code = 1
+
+raise SystemExit(exit_code)
+PY
+}
+
+section "Policy"
+if [ -s "$BUDGET_FILE" ]; then
+    printf 'budget_file=%s\n' "$BUDGET_FILE"
+else
+    fail "budget file is missing: $BUDGET_FILE"
+fi
+
+host_budget
+kubernetes_budget || failures=$((failures + 1))
+
+section "Summary"
+printf 'failures=%s warnings=%s\n' "$failures" "$warnings"
+
+if [ "$failures" -gt 0 ]; then
+    exit 1
+fi

scripts/restore-drill

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+KUBECONFIG="${KUBECONFIG:-${KUBECONFIG_PATH:-${HOME}/.kube/config}}"
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+check_file() {
+    local label="$1"
+    local path="$2"
+
+    printf '%-34s ' "${label}"
+    if [[ -s "${path}" ]]; then
+        printf 'ok - %s\n' "${path}"
+    else
+        printf 'fail - missing or empty: %s\n' "${path}"
+        return 1
+    fi
+}
+
+latest_file() {
+    local directory="$1"
+    local glob="$2"
+
+    find "${directory}" -maxdepth 1 -type f -name "${glob}" -print0 2>/dev/null |
+        xargs -0 ls -t 2>/dev/null |
+        sed -n '1p'
+}
+
+drill_gitea() {
+    local backup_dir="${LAB_GITEA_BACKUP_DIR:-/home/jv/backups/gitea}"
+    local latest
+
+    section "Gitea Restore Drill"
+    latest="$(latest_file "${backup_dir}" 'gitea-*.zip' || true)"
+    if [[ -z "${latest}" ]]; then
+        printf 'fail - no Gitea backup archive found in %s\n' "${backup_dir}"
+        return 1
+    fi
+
+    python3 - "${latest}" <<'PY'
+import sys
+import zipfile
+
+archive = sys.argv[1]
+with zipfile.ZipFile(archive) as handle:
+    bad = handle.testzip()
+    if bad:
+        raise SystemExit(f"fail - corrupt archive member: {bad}")
+    members = handle.namelist()
+    if not members:
+        raise SystemExit("fail - archive is empty")
+print(f"ok - archive valid: {archive} ({len(members)} members)")
+PY
+}
+
+drill_pihole() {
+    section "Pi-hole Restore Drill"
+    check_file "adlists" "${REPO_ROOT}/infra/rpi-services/adlists.txt"
+    check_file "local DNS records" "${REPO_ROOT}/infra/rpi-services/local-dns-records.txt"
+    check_file "CNAME records" "${REPO_ROOT}/infra/rpi-services/cname-records.txt"
+    check_file "static DHCP notes" "${REPO_ROOT}/infra/rpi-services/static-dhcp-hosts.txt"
+    check_file "RPi compose" "${REPO_ROOT}/infra/rpi-services/docker-compose.yml"
+    check_file "RPi bootstrap" "${REPO_ROOT}/infra/rpi-services/bootstrap.sh"
+}
+
+drill_state() {
+    local backup_dir="${HOMELAB_TOFU_STATE_BACKUP_DIR:-${HOMELAB_STATE_DIR}/tofu-state-backups}"
+    local latest
+
+    section "Jeannie/OpenTofu State Restore Drill"
+    latest="$(latest_file "${backup_dir}" 'tofu-state-*.tgz' || true)"
+    if [[ -z "${latest}" ]]; then
+        printf 'warn - no local OpenTofu state backup archive found in %s\n' "${backup_dir}"
+        printf 'hint - run ./jeannie state-backup after important changes\n'
+        return 0
+    fi
+    tar -tzf "${latest}" >/dev/null
+    printf 'ok - state archive can be listed: %s\n' "${latest}"
+}
+
+drill_gitops() {
+    section "GitOps Rebuild Drill"
+    git -C "${REPO_ROOT}" status --short
+    git -C "${REPO_ROOT}" rev-parse --verify HEAD >/dev/null
+    printf 'ok - repository HEAD: %s\n' "$(git -C "${REPO_ROOT}" rev-parse --short HEAD)"
+
+    if command -v kubectl >/dev/null 2>&1 && [[ -s "${KUBECONFIG}" ]]; then
+        kubectl --kubeconfig "${KUBECONFIG}" -n argocd get applications.argoproj.io 2>/dev/null || true
+    else
+        printf 'info - kubeconfig unavailable; skipping Argo CD application list\n'
+    fi
+}
+
+usage() {
+    cat <<'EOF'
+Usage: scripts/restore-drill {all|gitea|pihole|state|gitops}
+
+Read-only restore drills. These checks prove that restore inputs exist and can
+be inspected, but they do not replace running services or mutate Kubernetes.
+EOF
+}
+
+main() {
+    case "${1:-all}" in
+        all)
+            drill_gitea
+            drill_pihole
+            drill_state
+            drill_gitops
+            ;;
+        gitea)
+            drill_gitea
+            ;;
+        pihole)
+            drill_pihole
+            ;;
+        state)
+            drill_state
+            ;;
+        gitops)
+            drill_gitops
+            ;;
+        -h | --help | help)
+            usage
+            ;;
+        *)
+            usage >&2
+            return 2
+            ;;
+    esac
+}
+
+main "$@"

scripts/route-inventory

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-/home/jv/.kube/config}}"
+KUMA_MONITORS_FILE="${LAB_UPTIME_KUMA_MONITORS_FILE:-infra/rpi-services/uptime-kuma-monitors.json}"
+INGRESS_JSON=""
+
+cleanup() {
+    if [ -n "$INGRESS_JSON" ] && [ -f "$INGRESS_JSON" ]; then
+        rm -f "$INGRESS_JSON"
+    fi
+}
+trap cleanup EXIT
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+if ! command -v kubectl >/dev/null 2>&1 || [ ! -s "$KUBECONFIG_PATH" ]; then
+    echo "kubectl and kubeconfig are required for route-inventory." >&2
+    exit 1
+fi
+if ! kubectl --kubeconfig "$KUBECONFIG_PATH" get --raw=/readyz >/dev/null 2>&1; then
+    echo "Kubernetes API is unavailable." >&2
+    exit 1
+fi
+
+section "Ingress Routes"
+INGRESS_JSON="$(mktemp "${TMPDIR:-/tmp}/homelab-ingress.XXXXXX")"
+kubectl --kubeconfig "$KUBECONFIG_PATH" get ingress -A -o json >"$INGRESS_JSON"
+python3 - "$INGRESS_JSON" "$KUMA_MONITORS_FILE" <<'PY'
+import json
+import sys
+
+ingress_file, monitors_file = sys.argv[1:3]
+with open(ingress_file, encoding="utf-8") as handle:
+    doc = json.load(handle)
+monitors = []
+try:
+    with open(monitors_file, encoding="utf-8") as handle:
+        data = json.load(handle)
+    for item in data if isinstance(data, list) else data.get("monitors", []):
+        url = item.get("url") or item.get("hostname") or ""
+        monitors.append(url)
+except (OSError, json.JSONDecodeError):
+    pass
+
+print(f"{'NAMESPACE':20} {'NAME':28} {'HOST':36} {'PATH':16} {'SERVICE':28} {'TLS':6} {'KUMA':6}")
+missing = []
+for ingress in doc.get("items", []):
+    ns = ingress["metadata"]["namespace"]
+    name = ingress["metadata"]["name"]
+    tls_hosts = {
+        host
+        for tls in ingress.get("spec", {}).get("tls", [])
+        for host in tls.get("hosts", [])
+    }
+    for rule in ingress.get("spec", {}).get("rules", []):
+        host = rule.get("host", "")
+        paths = rule.get("http", {}).get("paths", [])
+        if not paths:
+            paths = [{"path": "/", "backend": {}}]
+        for path in paths:
+            backend = path.get("backend", {}).get("service", {})
+            service = backend.get("name", "")
+            port = backend.get("port", {}).get("number") or backend.get("port", {}).get("name") or ""
+            route_path = path.get("path", "/")
+            tls = "yes" if host in tls_hosts else "no"
+            kuma = "yes" if any(host and host in monitor for monitor in monitors) else "no"
+            print(f"{ns:20} {name:28} {host:36} {route_path:16} {(service + ':' + str(port)):28} {tls:6} {kuma:6}")
+            if kuma == "no":
+                missing.append(f"{host}{route_path}")
+
+if missing:
+    print("\nRoutes missing visible Uptime Kuma monitor coverage:")
+    for route in sorted(set(missing)):
+        print(f"  {route}")
+PY
+
+section "Services Behind Ingress"
+kubectl --kubeconfig "$KUBECONFIG_PATH" get svc -A \
+    -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,PORTS:.spec.ports[*].port' |
+    sed -n '1,120p'

scripts/scorecard

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+failures=0
+warnings=0
+
+score() {
+    local label="$1"
+    local severity="$2"
+    shift 2
+    local output
+    local summary
+    local status
+
+    printf '%-28s ' "$label"
+    set +e
+    output="$("$@" 2>&1)"
+    status=$?
+    set -e
+    if [ "$status" -eq 0 ]; then
+        printf 'pass\n'
+        return 0
+    fi
+    if [ "$severity" = "fail" ]; then
+        failures=$((failures + 1))
+        printf 'fail'
+    else
+        warnings=$((warnings + 1))
+        printf 'warn'
+    fi
+    summary="$(
+        printf '%s\n' "$output" |
+            awk 'NF && $0 !~ /^\+/ { print; exit }'
+    )"
+    if [ -z "$summary" ]; then
+        summary="check exited with status ${status}"
+    fi
+    printf ' - %s' "$summary"
+    printf '\n'
+}
+
+docs_fresh() {
+    "${REPO_ROOT}/scripts/render-docs" --check >/dev/null
+}
+
+inventory_ok() {
+    "${REPO_ROOT}/scripts/validate-homelab-inventory" "${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}" >/dev/null
+}
+
+security_static_ok() {
+    "${REPO_ROOT}/scripts/validate-tailnet-policy" >/dev/null
+}
+
+printf 'Homelab scorecard\n'
+printf '=================\n\n'
+
+score "Inventory" fail inventory_ok
+score "Documentation freshness" warn docs_fresh
+score "Backup readiness" fail "${REPO_ROOT}/scripts/backup-status"
+score "GitOps health" warn "${REPO_ROOT}/scripts/gitops-status"
+score "DNS and RPi health" fail "${REPO_ROOT}/jeannie" doctor-rpi
+score "Cluster health" fail "${REPO_ROOT}/jeannie" doctor-cluster
+score "Edge health" fail "${REPO_ROOT}/jeannie" doctor-edge
+score "Capacity pressure" warn "${REPO_ROOT}/scripts/resource-budget"
+score "Security posture" warn security_static_ok
+score "Public certificates" warn "${REPO_ROOT}/scripts/cert-check"
+
+printf '\nSummary: failures=%s warnings=%s\n' "$failures" "$warnings"
+if [ "$failures" -gt 0 ]; then
+    printf 'Focus on the failed categories first, then clear warnings.\n'
+    exit 1
+fi
+if [ "$warnings" -gt 0 ]; then
+    printf 'No blocking failures, but warnings need cleanup.\n'
+    exit 0
+fi
+printf 'Scorecard passed.\n'

scripts/security-attack-path

@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG="${KUBECONFIG:-${HOME}/.kube/config}"
+ATTACK_NAMESPACE="${1:-${SECURITY_ATTACK_NAMESPACE:-website-production}}"
+ATTACK_WORKLOAD="${2:-${SECURITY_ATTACK_WORKLOAD:-deployment/php-website-deployment}}"
+
+usage() {
+    cat <<'EOF'
+Usage: scripts/security-attack-path [namespace] [workload]
+
+Default:
+  scripts/security-attack-path website-production deployment/php-website-deployment
+
+This is a read-only compromise-path drill. It inspects what a compromised pod
+would inherit: service account, mounted secrets, RBAC bindings, network
+policies, and nearby services.
+EOF
+}
+
+require_kubectl() {
+    if ! command -v kubectl >/dev/null 2>&1; then
+        echo "kubectl is required." >&2
+        exit 1
+    fi
+    if [[ ! -f "${KUBECONFIG}" ]]; then
+        echo "KUBECONFIG does not exist: ${KUBECONFIG}" >&2
+        exit 1
+    fi
+}
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+kubectl_ro() {
+    kubectl --kubeconfig "${KUBECONFIG}" "$@"
+}
+
+jsonpath_or_unknown() {
+    local resource="$1"
+    local path="$2"
+    local value
+
+    value="$(kubectl_ro -n "${ATTACK_NAMESPACE}" get "${resource}" -o "jsonpath=${path}" 2>/dev/null || true)"
+    printf '%s\n' "${value:-unknown}"
+}
+
+main() {
+    local service_account
+    local automount
+    local selector
+
+    case "${1:-}" in
+        -h | --help | help)
+            usage
+            return 0
+            ;;
+    esac
+
+    require_kubectl
+
+    section "Attack Path Drill"
+    printf 'Namespace: %s\n' "${ATTACK_NAMESPACE}"
+    printf 'Workload:  %s\n' "${ATTACK_WORKLOAD}"
+
+    if ! kubectl_ro -n "${ATTACK_NAMESPACE}" get "${ATTACK_WORKLOAD}" >/dev/null 2>&1; then
+        echo "Target workload not found." >&2
+        exit 1
+    fi
+
+    section "Identity"
+    service_account="$(jsonpath_or_unknown "${ATTACK_WORKLOAD}" '{.spec.template.spec.serviceAccountName}')"
+    automount="$(jsonpath_or_unknown "${ATTACK_WORKLOAD}" '{.spec.template.spec.automountServiceAccountToken}')"
+    service_account="${service_account:-default}"
+    printf 'ServiceAccount: %s\n' "${service_account}"
+    printf 'Automount token: %s\n' "${automount}"
+    if [[ "${automount}" != "false" ]]; then
+        printf 'Focus: pod may receive a Kubernetes API token unless blocked by defaults.\n'
+    fi
+
+    section "Mounted Secrets And Config"
+    kubectl_ro -n "${ATTACK_NAMESPACE}" get "${ATTACK_WORKLOAD}" \
+        -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" secret="}{.secret.secretName}{" configMap="}{.configMap.name}{" pvc="}{.persistentVolumeClaim.claimName}{" hostPath="}{.hostPath.path}{"\n"}{end}' || true
+
+    section "RBAC Bindings In Namespace"
+    kubectl_ro -n "${ATTACK_NAMESPACE}" get rolebinding -o wide 2>/dev/null || true
+    printf '\nClusterRoleBindings mentioning service account %s/%s:\n' "${ATTACK_NAMESPACE}" "${service_account}"
+    kubectl_ro get clusterrolebinding -o jsonpath='{range .items[?(@.subjects)]}{.metadata.name}{" "}{range .subjects[*]}{.kind}{"/"}{.namespace}{"/"}{.name}{" "}{end}{"\n"}{end}' 2>/dev/null |
+        grep -F "ServiceAccount/${ATTACK_NAMESPACE}/${service_account}" || true
+
+    section "Network Policies"
+    kubectl_ro -n "${ATTACK_NAMESPACE}" get networkpolicy -o wide 2>/dev/null || true
+    if ! kubectl_ro -n "${ATTACK_NAMESPACE}" get networkpolicy >/dev/null 2>&1; then
+        printf 'Focus: no NetworkPolicy objects found; pod egress is probably unrestricted by namespace policy.\n'
+    fi
+
+    section "Nearby Services"
+    kubectl_ro -n "${ATTACK_NAMESPACE}" get svc,endpoints -o wide 2>/dev/null || true
+
+    section "Selected Pods"
+    # shellcheck disable=SC2016
+    selector="$(kubectl_ro -n "${ATTACK_NAMESPACE}" get "${ATTACK_WORKLOAD}" -o go-template='{{range $k,$v := .spec.selector.matchLabels}}{{printf "%s=%s," $k $v}}{{end}}' 2>/dev/null | sed 's/,$//')"
+    if [[ -n "${selector}" ]]; then
+        kubectl_ro -n "${ATTACK_NAMESPACE}" get pods -l "${selector}" -o wide
+    else
+        kubectl_ro -n "${ATTACK_NAMESPACE}" get pods -o wide
+    fi
+
+    section "Manual Follow-Up"
+    cat <<EOF
+Ask these questions before adding a fix:
+- Does the pod need a service account token?
+- Are any mounted Secrets unnecessary for runtime?
+- Can this namespace use default-deny egress with explicit allows?
+- Can the workload run as non-root with allowPrivilegeEscalation=false?
+- Should this namespace be isolated from Gitea, registry, Pi-hole, and Kubernetes API?
+EOF
+}
+
+main "$@"

scripts/security-logs

@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+SECURITY_REPORT_DIR="${SECURITY_REPORT_DIR:-${HOMELAB_STATE_DIR}/security-reports}"
+SECURITY_LOG_SINCE="${SECURITY_LOG_SINCE:-24h}"
+GITEA_CONTAINER="${GITEA_CONTAINER:-${LAB_GITEA_CONTAINER_NAME:-homelab-gitea}}"
+KUBECONFIG="${KUBECONFIG:-${HOME}/.kube/config}"
+
+timestamp() {
+    date -u +%Y%m%dT%H%M%SZ
+}
+
+ensure_report_dir() {
+    mkdir -p "${SECURITY_REPORT_DIR}"
+}
+
+print_matches() {
+    local title="$1"
+    local log_file="$2"
+    local pattern="$3"
+    local limit="${4:-20}"
+
+    printf '\n== %s ==\n' "${title}"
+    if [[ ! -s "${log_file}" ]]; then
+        printf 'no log input\n'
+        return 0
+    fi
+
+    if grep -Eai "${pattern}" "${log_file}" >/dev/null; then
+        grep -Eai "${pattern}" "${log_file}" | tail -n "${limit}"
+    else
+        printf 'none\n'
+    fi
+}
+
+collect_gitea_logs() {
+    local output_file="$1"
+
+    if ! command -v docker >/dev/null 2>&1; then
+        return 0
+    fi
+    if ! docker ps --format '{{.Names}}' | grep -Fxq "${GITEA_CONTAINER}"; then
+        return 0
+    fi
+
+    docker logs --since "${SECURITY_LOG_SINCE}" "${GITEA_CONTAINER}" >"${output_file}" 2>&1 || true
+}
+
+collect_traefik_logs() {
+    local output_file="$1"
+
+    if ! command -v kubectl >/dev/null 2>&1; then
+        return 0
+    fi
+    if [[ ! -f "${KUBECONFIG}" ]]; then
+        return 0
+    fi
+
+    kubectl --kubeconfig "${KUBECONFIG}" -n traefik logs \
+        -l app.kubernetes.io/name=traefik \
+        --since="${SECURITY_LOG_SINCE}" \
+        --tail="${SECURITY_LOG_TAIL:-5000}" >"${output_file}" 2>&1 || true
+}
+
+main() {
+    local stamp
+    local gitea_log
+    local traefik_log
+    local report_file
+    local suspicious_paths
+    local suspicious_agents
+
+    stamp="$(timestamp)"
+    ensure_report_dir
+    gitea_log="${SECURITY_REPORT_DIR}/security-gitea-logs-${stamp}.txt"
+    traefik_log="${SECURITY_REPORT_DIR}/security-traefik-logs-${stamp}.txt"
+    report_file="${SECURITY_REPORT_DIR}/security-logs-${stamp}.txt"
+    suspicious_paths='/(wp-admin|wp-login|xmlrpc\.php|\.env|\.git|vendor/phpunit|cgi-bin|boaform|solr|actuator|server-status)|\.\./|%2e%2e|/etc/passwd|/proc/self/environ'
+    suspicious_agents='nikto|sqlmap|nuclei|masscan|zgrab|gobuster|dirbuster|dirsearch|wpscan|acunetix|nessus|openvas|curl/[0-9]|python-requests|Go-http-client'
+
+    collect_gitea_logs "${gitea_log}"
+    collect_traefik_logs "${traefik_log}"
+
+    {
+        echo "Homelab security log summary"
+        echo "Generated: ${stamp}"
+        echo "Window: ${SECURITY_LOG_SINCE}"
+        echo
+        print_matches "Gitea auth or permission failures" "${gitea_log}" 'authentication failed|invalid credentials|denied|forbidden|unauthorized|failed login|401|403'
+        print_matches "Gitea suspicious repo/path activity" "${gitea_log}" "${suspicious_paths}"
+        print_matches "Traefik suspicious paths" "${traefik_log}" "${suspicious_paths}"
+        print_matches "Traefik suspicious user agents" "${traefik_log}" "${suspicious_agents}"
+        print_matches "Traefik client/server errors" "${traefik_log}" ' 4[0-9][0-9] | 5[0-9][0-9] |level=error|level=warn'
+        echo
+        echo "Raw Gitea log sample: ${gitea_log}"
+        echo "Raw Traefik log sample: ${traefik_log}"
+    } | tee "${report_file}"
+
+    echo "Security log report: ${report_file}"
+}
+
+main "$@"

scripts/security-scan

@@ -0,0 +1,551 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+HOMELAB_STATE_DIR="${HOMELAB_STATE_DIR:-${XDG_DATA_HOME:-${HOME}/.local/share}/homelab}"
+SECURITY_REPORT_DIR="${SECURITY_REPORT_DIR:-${HOMELAB_STATE_DIR}/security-reports}"
+SECURITY_TARGETS_FILE="${SECURITY_TARGETS_FILE:-${REPO_ROOT}/security/targets.txt}"
+TRIVY_IMAGE="${TRIVY_IMAGE:-aquasec/trivy:latest}"
+ZAP_IMAGE="${ZAP_IMAGE:-ghcr.io/zaproxy/zaproxy:stable}"
+KUBE_BENCH_IMAGE="${KUBE_BENCH_IMAGE:-aquasec/kube-bench:latest}"
+NUCLEI_IMAGE="${NUCLEI_IMAGE:-projectdiscovery/nuclei:latest}"
+GITLEAKS_IMAGE="${GITLEAKS_IMAGE:-ghcr.io/gitleaks/gitleaks:latest}"
+
+truthy() {
+    case "${1,,}" in
+        1 | true | yes | y | on | enabled)
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+usage() {
+    cat <<'EOF'
+Usage: scripts/security-scan {all|prepare|trivy|zap|kube-bench|nuclei|host|web|secrets}
+
+Environment:
+  SECURITY_REPORT_DIR    Report output directory.
+  SECURITY_TARGETS_FILE  Target allowlist for web scans.
+  SECURITY_ZAP_TARGET    Single URL override for ZAP.
+  SECURITY_NUCLEI_TARGET Single URL override for nuclei.
+  SECURITY_WEB_TARGET    Single URL override for web header checks.
+  LAB_BACKSTAGE_BRAIN_ENABLED=true enables LLM-assisted Lynis triage.
+EOF
+}
+
+timestamp() {
+    date -u +%Y%m%dT%H%M%SZ
+}
+
+ensure_report_dir() {
+    mkdir -p "${SECURITY_REPORT_DIR}"
+}
+
+require_docker() {
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "docker is required for containerized security scans." >&2
+        exit 1
+    fi
+}
+
+prepare_security_scans() {
+    local image
+    local -a images=(
+        "${TRIVY_IMAGE}"
+        "${ZAP_IMAGE}"
+        "${KUBE_BENCH_IMAGE}"
+        "${NUCLEI_IMAGE}"
+        "${GITLEAKS_IMAGE}"
+    )
+
+    ensure_report_dir
+    require_docker
+
+    if ! docker info >/dev/null 2>&1; then
+        echo "docker is installed but not reachable by this user." >&2
+        exit 1
+    fi
+    if [[ ! -s "${SECURITY_TARGETS_FILE}" ]]; then
+        echo "Missing target file: ${SECURITY_TARGETS_FILE}" >&2
+        exit 1
+    fi
+    if [[ ! -w "${SECURITY_REPORT_DIR}" ]]; then
+        echo "Report directory is not writable: ${SECURITY_REPORT_DIR}" >&2
+        exit 1
+    fi
+
+    echo "Preparing security scan images..."
+    for image in "${images[@]}"; do
+        echo "--> docker pull ${image}"
+        docker pull "${image}"
+    done
+
+    echo "Security scan prerequisites are ready."
+    echo "Report directory: ${SECURITY_REPORT_DIR}"
+}
+
+run_trivy() {
+    local stamp
+    stamp="$(timestamp)"
+    ensure_report_dir
+
+    echo "Running Trivy filesystem and IaC/config scans..."
+    if command -v trivy >/dev/null 2>&1; then
+        trivy fs --scanners vuln,secret,misconfig --format table --output "${SECURITY_REPORT_DIR}/trivy-fs-${stamp}.txt" "${REPO_ROOT}"
+        trivy config --format table --output "${SECURITY_REPORT_DIR}/trivy-config-${stamp}.txt" "${REPO_ROOT}"
+    else
+        require_docker
+        docker run --rm \
+            -v "${REPO_ROOT}:/repo:ro" \
+            -v "${SECURITY_REPORT_DIR}:/reports" \
+            "${TRIVY_IMAGE}" fs --scanners vuln,secret,misconfig --format table --output "/reports/trivy-fs-${stamp}.txt" /repo
+        docker run --rm \
+            -v "${REPO_ROOT}:/repo:ro" \
+            -v "${SECURITY_REPORT_DIR}:/reports" \
+            "${TRIVY_IMAGE}" config --format table --output "/reports/trivy-config-${stamp}.txt" /repo
+    fi
+
+    echo "Trivy reports written to ${SECURITY_REPORT_DIR}"
+}
+
+run_gitleaks() {
+    local stamp
+    local report_file
+    stamp="$(timestamp)"
+    ensure_report_dir
+    report_file="${SECURITY_REPORT_DIR}/gitleaks-${stamp}.json"
+
+    echo "Running gitleaks secret scan against this repo..."
+    if command -v gitleaks >/dev/null 2>&1; then
+        gitleaks detect \
+            --source "${REPO_ROOT}" \
+            --report-format json \
+            --report-path "${report_file}" \
+            --redact
+    else
+        require_docker
+        docker run --rm \
+            -v "${REPO_ROOT}:/repo:ro" \
+            -v "${SECURITY_REPORT_DIR}:/reports:rw" \
+            "${GITLEAKS_IMAGE}" detect \
+            --source /repo \
+            --report-format json \
+            --report-path "/reports/gitleaks-${stamp}.json" \
+            --redact
+    fi
+
+    echo "gitleaks report: ${report_file}"
+}
+
+scan_targets() {
+    local target_override="$1"
+
+    if [[ -n "${target_override}" ]]; then
+        printf '%s\n' "${target_override}"
+        return 0
+    fi
+
+    if [[ ! -s "${SECURITY_TARGETS_FILE}" ]]; then
+        echo "Missing target file: ${SECURITY_TARGETS_FILE}" >&2
+        exit 1
+    fi
+
+    sed -e 's/#.*//' -e '/^[[:space:]]*$/d' "${SECURITY_TARGETS_FILE}"
+}
+
+safe_target_name() {
+    printf '%s' "$1" | sed -E 's#^https?://##; s#[^A-Za-z0-9._-]+#_#g; s#_+$##'
+}
+
+run_zap() {
+    local stamp
+    local target
+    local safe_name
+    stamp="$(timestamp)"
+    ensure_report_dir
+    require_docker
+
+    echo "Running OWASP ZAP baseline passive scans..."
+    while IFS= read -r target; do
+        [[ -n "${target}" ]] || continue
+        safe_name="$(safe_target_name "${target}")"
+        echo "--> ZAP baseline ${target}"
+        docker run --rm \
+            -v "${SECURITY_REPORT_DIR}:/zap/wrk:rw" \
+            "${ZAP_IMAGE}" zap-baseline.py \
+            -t "${target}" \
+            -I \
+            -r "zap-${safe_name}-${stamp}.html" \
+            -J "zap-${safe_name}-${stamp}.json"
+    done < <(scan_targets "${SECURITY_ZAP_TARGET:-}")
+
+    echo "ZAP reports written to ${SECURITY_REPORT_DIR}"
+}
+
+run_nuclei() {
+    local stamp
+    local target
+    local safe_name
+    stamp="$(timestamp)"
+    ensure_report_dir
+    require_docker
+
+    echo "Running nuclei low-noise HTTP exposure scans..."
+    while IFS= read -r target; do
+        [[ -n "${target}" ]] || continue
+        safe_name="$(safe_target_name "${target}")"
+        echo "--> nuclei ${target}"
+        docker run --rm \
+            -v "${SECURITY_REPORT_DIR}:/reports:rw" \
+            "${NUCLEI_IMAGE}" \
+            -u "${target}" \
+            -severity low,medium,high,critical \
+            -rate-limit "${SECURITY_NUCLEI_RATE_LIMIT:-5}" \
+            -retries 1 \
+            -timeout 5 \
+            -o "/reports/nuclei-${safe_name}-${stamp}.txt"
+    done < <(scan_targets "${SECURITY_NUCLEI_TARGET:-}")
+
+    echo "nuclei reports written to ${SECURITY_REPORT_DIR}"
+}
+
+header_value() {
+    local headers_file="$1"
+    local header_name="$2"
+
+    awk -v name="${header_name}" '
+        BEGIN { IGNORECASE = 1 }
+        index($0, name ":") == 1 {
+            sub(/^[^:]+:[[:space:]]*/, "")
+            sub(/\r$/, "")
+            print
+            exit
+        }
+    ' "${headers_file}"
+}
+
+run_security_web() {
+    local stamp
+    local target
+    local safe_name
+    local headers_file
+    local body_file
+    local report_file
+    local status
+    local failures=0
+    local missing=0
+    local -a required_headers=(
+        "strict-transport-security"
+        "x-content-type-options"
+        "x-frame-options"
+        "referrer-policy"
+    )
+
+    stamp="$(timestamp)"
+    ensure_report_dir
+    report_file="${SECURITY_REPORT_DIR}/security-web-${stamp}.txt"
+
+    {
+        echo "Security web quick check"
+        echo "Generated: ${stamp}"
+        echo
+    } >"${report_file}"
+
+    while IFS= read -r target; do
+        [[ -n "${target}" ]] || continue
+        safe_name="$(safe_target_name "${target}")"
+        headers_file="${SECURITY_REPORT_DIR}/headers-${safe_name}-${stamp}.txt"
+        body_file="${SECURITY_REPORT_DIR}/headers-${safe_name}-${stamp}.body"
+
+        echo "== ${target}" | tee -a "${report_file}"
+        if [[ "${target}" != https://* ]]; then
+            echo "fail: target is not HTTPS" | tee -a "${report_file}"
+            failures=$((failures + 1))
+        fi
+
+        status="$(
+            curl -k -sS -L \
+                --connect-timeout "${SECURITY_WEB_CONNECT_TIMEOUT:-8}" \
+                --max-time "${SECURITY_WEB_MAX_TIME:-20}" \
+                -D "${headers_file}" \
+                -o "${body_file}" \
+                -w '%{http_code}' \
+                "${target}" 2>>"${report_file}" || true
+        )"
+
+        if [[ ! "${status}" =~ ^2|3[0-9][0-9]$ ]]; then
+            echo "fail: HTTP status ${status:-curl failed}" | tee -a "${report_file}"
+            failures=$((failures + 1))
+        else
+            echo "ok: HTTP status ${status}" | tee -a "${report_file}"
+        fi
+
+        missing=0
+        for header_name in "${required_headers[@]}"; do
+            if [[ -n "$(header_value "${headers_file}" "${header_name}")" ]]; then
+                echo "ok: ${header_name}" | tee -a "${report_file}"
+            else
+                echo "warn: missing ${header_name}" | tee -a "${report_file}"
+                missing=$((missing + 1))
+            fi
+        done
+
+        if [[ -n "$(header_value "${headers_file}" "content-security-policy")" ]]; then
+            echo "ok: content-security-policy" | tee -a "${report_file}"
+        else
+            echo "info: content-security-policy missing or intentionally deferred" | tee -a "${report_file}"
+        fi
+
+        if ((missing > 0)); then
+            failures=$((failures + 1))
+        fi
+        echo | tee -a "${report_file}" >/dev/null
+        rm -f "${body_file}"
+    done < <(scan_targets "${SECURITY_WEB_TARGET:-}")
+
+    echo "Security web report: ${report_file}"
+    if ((failures > 0)); then
+        echo "security-web found ${failures} target(s) needing review." >&2
+        return 1
+    fi
+}
+
+run_kube_bench() {
+    local stamp
+    stamp="$(timestamp)"
+    ensure_report_dir
+    require_docker
+
+    echo "Running kube-bench CIS checks against this kubeadm host..."
+    docker run --rm --pid=host \
+        -v /etc:/etc:ro \
+        -v /var:/var:ro \
+        -v /usr/bin:/usr/local/mount-from-host/bin:ro \
+        "${KUBE_BENCH_IMAGE}" run --targets master,node | tee "${SECURITY_REPORT_DIR}/kube-bench-${stamp}.txt"
+}
+
+lynis_findings() {
+    local report_file="$1"
+
+    sed -n 's/^warning\[\]=/warning|/p; s/^suggestion\[\]=/suggestion|/p' "${report_file}" 2>/dev/null |
+        awk -F'|' 'NF >= 2 { source = $1; finding = $2; test = $NF; print source "|" finding (NF > 2 ? " [" test "]" : "") }'
+}
+
+print_lynis_fallback_focus() {
+    local findings_file="$1"
+    local selected
+
+    selected="$(awk -F'|' '$1 == "warning" { print; exit }' "${findings_file}")"
+    if [[ -z "${selected}" ]]; then
+        selected="$(awk -F'|' '$1 == "suggestion" { print; exit }' "${findings_file}")"
+    fi
+
+    if [[ -z "${selected}" ]]; then
+        printf '\n== Highest-Risk Lynis Finding ==\n'
+        printf 'No Lynis warnings or suggestions were found.\n'
+        return 0
+    fi
+
+    printf '\n== Highest-Risk Lynis Finding ==\n'
+    printf 'Source: %s\n' "$(cut -d'|' -f1 <<<"${selected}")"
+    printf 'Finding: %s\n' "$(cut -d'|' -f2- <<<"${selected}")"
+    printf 'Why this one: LLM triage was unavailable, so Jeannie selected the first Lynis warning, falling back to the first suggestion.\n'
+    printf 'Mode: report-only. Add an explicit narrow Jeannie hardening command before changing host state.\n'
+}
+
+print_lynis_llm_focus() {
+    local findings_file="$1"
+    local endpoint="${LAB_AI_GATEWAY_URL:-${LAB_OLLAMA_URL:-http://127.0.0.1:11434}}"
+    local model="${LAB_AI_GATEWAY_MODEL:-qwen2.5:0.5b}"
+    local timeout="${LAB_AI_GATEWAY_TIMEOUT_SECONDS:-20}"
+    local index_dir="${LAB_AI_KNOWLEDGE_INDEX_DIR:-/data/homelab-ai/index}"
+    local context=""
+
+    if ! truthy "${LAB_BACKSTAGE_BRAIN_ENABLED:-false}"; then
+        return 1
+    fi
+    if ! command -v python3 >/dev/null 2>&1; then
+        return 1
+    fi
+
+    if [[ -s "${index_dir}/index.json" && -x "${REPO_ROOT}/scripts/query-homelab-ai-index" ]]; then
+        context="$("${REPO_ROOT}/scripts/query-homelab-ai-index" --index-dir "${index_dir}" --context-only --limit 4 "Lynis host hardening Debian Docker kubeadm SSH homelab" 2>/dev/null || true)"
+    fi
+
+    LYNIS_FINDINGS="$(head -120 "${findings_file}")" \
+        LYNIS_CONTEXT="${context}" \
+        python3 - "${endpoint}" "${model}" "${timeout}" <<'PY'
+import json
+import os
+import sys
+import urllib.error
+import urllib.request
+
+endpoint, model, timeout = sys.argv[1:4]
+findings = os.environ.get("LYNIS_FINDINGS", "").strip()
+context = os.environ.get("LYNIS_CONTEXT", "").strip()
+if not findings:
+    raise SystemExit(1)
+
+try:
+    timeout_seconds = int(timeout)
+except ValueError:
+    timeout_seconds = 20
+
+try:
+    with urllib.request.urlopen(f"{endpoint.rstrip('/')}/api/tags", timeout=3) as response:
+        tags = json.loads(response.read().decode("utf-8"))
+    available = {item.get("name", "") for item in tags.get("models", [])}
+    if model not in available and f"{model}:latest" not in available:
+        raise SystemExit(1)
+except (OSError, TimeoutError, urllib.error.URLError, json.JSONDecodeError):
+    raise SystemExit(1)
+
+prompt = f"""You are Jeannie, a report-only homelab security triage helper.
+The host is a Debian personal homelab server running Docker, kubeadm, Gitea,
+Tailscale, local registry, and public edge routing. Do not recommend broad
+automatic fixes. Pick exactly ONE Lynis finding to focus on first.
+
+Rank by risk and operational value. Prefer findings that reduce real exposure
+without likely breaking SSH, Docker networking, Kubernetes, or remote access.
+
+Return exactly this format:
+Finding: <one selected finding>
+Why highest risk: <one sentence>
+Next safe action: <one concrete manual validation or narrow remediation>
+Automation stance: report-only; add an explicit Jeannie command only after review
+
+Homelab context:
+{context}
+
+Lynis findings:
+{findings}
+"""
+
+payload = {
+    "model": model,
+    "prompt": prompt,
+    "stream": False,
+    "options": {
+        "temperature": 0.1,
+        "num_predict": 220,
+    },
+}
+
+try:
+    request = urllib.request.Request(
+        f"{endpoint.rstrip('/')}/api/generate",
+        data=json.dumps(payload).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
+        result = json.loads(response.read().decode("utf-8"))
+except (OSError, TimeoutError, urllib.error.URLError, json.JSONDecodeError):
+    raise SystemExit(1)
+
+answer = (result.get("response") or "").strip()
+if not answer:
+    raise SystemExit(1)
+
+print("\n== Highest-Risk Lynis Finding ==")
+print(answer)
+PY
+}
+
+print_lynis_focus() {
+    local report_file="$1"
+    local findings_file
+    local hardening_index
+
+    hardening_index="$(awk -F= '$1 == "hardening_index" { print $2; exit }' "${report_file}" 2>/dev/null || true)"
+
+    printf '\n== Lynis Host Audit ==\n'
+    if [[ -n "${hardening_index}" ]]; then
+        printf 'Hardening index: %s\n' "${hardening_index}"
+    fi
+
+    findings_file="$(mktemp)"
+    lynis_findings "${report_file}" >"${findings_file}"
+    if ! print_lynis_llm_focus "${findings_file}"; then
+        print_lynis_fallback_focus "${findings_file}"
+    fi
+    rm -f "${findings_file}"
+}
+
+run_lynis_host() {
+    local stamp
+    local report_file
+    local log_file
+    stamp="$(timestamp)"
+    ensure_report_dir
+    report_file="${SECURITY_REPORT_DIR}/lynis-host-${stamp}.dat"
+    log_file="${SECURITY_REPORT_DIR}/lynis-host-${stamp}.log"
+
+    if ! command -v lynis >/dev/null 2>&1; then
+        cat >&2 <<'EOF'
+lynis is not installed on this host.
+
+Install it on Debian with:
+
+  sudo apt install lynis
+
+Jeannie does not install Lynis automatically because host hardening audits should
+remain explicit and report-only.
+EOF
+        exit 1
+    fi
+
+    echo "Running Lynis host audit in report-only mode..."
+    sudo lynis audit system \
+        --no-colors \
+        --quiet \
+        --report-file "${report_file}" \
+        --log-file "${log_file}" >/dev/null
+
+    print_lynis_focus "${report_file}"
+    printf '\nFull Lynis report: %s\n' "${report_file}"
+    printf 'Full Lynis log:    %s\n' "${log_file}"
+}
+
+case "${1:-all}" in
+    all)
+        run_trivy
+        run_gitleaks
+        run_zap
+        ;;
+    prepare)
+        prepare_security_scans
+        ;;
+    trivy)
+        run_trivy
+        ;;
+    secrets)
+        run_gitleaks
+        ;;
+    zap)
+        run_zap
+        ;;
+    nuclei)
+        run_nuclei
+        ;;
+    web)
+        run_security_web
+        ;;
+    kube-bench)
+        run_kube_bench
+        ;;
+    host)
+        run_lynis_host
+        ;;
+    -h | --help | help)
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

scripts/synthetic-checks

@@ -0,0 +1,155 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-/home/jv/.kube/config}}"
+GITEA_HTTP_URL="${LAB_GITEA_ROOT_URL:-https://lab2025.duckdns.org/git/}"
+GITEA_LOCAL_URL="${LAB_GITEA_LOCAL_URL:-http://127.0.0.1:3000/}"
+GITEA_SSH_REMOTE="${LAB_GITOPS_REPO_URL:-ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git}"
+REGISTRY_ENDPOINT="${LAB_REGISTRY_ENDPOINT:-192.168.100.73:30500}"
+PIHOLE_HOST="${LAB_RPI_HOST:-192.168.100.89}"
+TRAEFIK_LB_IP="${LAB_TRAEFIK_LB_IP:-192.168.100.240}"
+PUBLIC_URL="${LAB_PUBLIC_URL:-https://lab2025.duckdns.org/}"
+
+failures=0
+warnings=0
+
+section() {
+    printf '\n== %s ==\n' "$1"
+}
+
+check() {
+    local label="$1"
+    shift
+    local output
+    local status
+
+    printf '%-34s ' "$label"
+    set +e
+    output="$("$@" 2>&1)"
+    status=$?
+    set -e
+    if [ "$status" -eq 0 ]; then
+        printf 'ok'
+        if [ -n "$output" ]; then
+            printf ' - %s' "$(printf '%s' "$output" | head -n 1)"
+        fi
+        printf '\n'
+    else
+        printf 'fail - %s\n' "$(printf '%s' "$output" | head -n 1)"
+        failures=$((failures + 1))
+    fi
+}
+
+warn_check() {
+    local label="$1"
+    shift
+    local output
+    local status
+
+    printf '%-34s ' "$label"
+    set +e
+    output="$("$@" 2>&1)"
+    status=$?
+    set -e
+    if [ "$status" -eq 0 ]; then
+        printf 'ok'
+        if [ -n "$output" ]; then
+            printf ' - %s' "$(printf '%s' "$output" | head -n 1)"
+        fi
+        printf '\n'
+    else
+        printf 'warn - %s\n' "$(printf '%s' "$output" | head -n 1)"
+        warnings=$((warnings + 1))
+    fi
+}
+
+http_ok() {
+    local url="$1"
+    local status
+
+    status="$(curl -k -sS -o /dev/null -w '%{http_code}' --max-time 10 "$url")"
+    case "$status" in
+        200|301|302|307|308|401|403|404)
+            echo "http=$status"
+            ;;
+        *)
+            echo "http=$status"
+            return 1
+            ;;
+    esac
+}
+
+dns_query() {
+    local server="$1"
+    local name="$2"
+
+    if command -v dig >/dev/null 2>&1; then
+        dig "@$server" "$name" A +time=3 +tries=1 +short | grep -q .
+    else
+        python3 - "$server" "$name" <<'PY'
+import random
+import socket
+import struct
+import sys
+
+server, name = sys.argv[1], sys.argv[2].rstrip(".")
+query_id = random.randrange(0, 65536)
+packet = struct.pack("!HHHHHH", query_id, 0x0100, 1, 0, 0, 0)
+for label in name.split("."):
+    packet += bytes([len(label)]) + label.encode("ascii")
+packet += b"\x00" + struct.pack("!HH", 1, 1)
+sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+sock.settimeout(5)
+sock.sendto(packet, (server, 53))
+data, _ = sock.recvfrom(512)
+if len(data) < 12:
+    raise SystemExit("short DNS response")
+_response_id, flags, _qd, answers, _ns, _ar = struct.unpack("!HHHHHH", data[:12])
+if flags & 0x000f:
+    raise SystemExit(f"dns rcode {flags & 0x000f}")
+if answers < 1:
+    raise SystemExit("no answers")
+PY
+    fi
+}
+
+kubectl_ok() {
+    command -v kubectl >/dev/null 2>&1 && [ -s "$KUBECONFIG_PATH" ] &&
+        kubectl --kubeconfig "$KUBECONFIG_PATH" get --raw=/readyz >/dev/null 2>&1
+}
+
+section "Gitea And Git"
+check "Gitea public HTTP" http_ok "$GITEA_HTTP_URL"
+check "Gitea local HTTP" http_ok "$GITEA_LOCAL_URL"
+warn_check "Gitea SSH ls-remote" git ls-remote "$GITEA_SSH_REMOTE" HEAD
+
+section "Registry"
+check "Registry API" http_ok "http://${REGISTRY_ENDPOINT}/v2/"
+if [ "${LAB_SYNTHETIC_REGISTRY_PUSH:-false}" = "true" ]; then
+    check "Registry push/pull smoke" sh -c "docker pull busybox:latest >/dev/null && docker tag busybox:latest '${REGISTRY_ENDPOINT}/synthetic/busybox:latest' && docker push '${REGISTRY_ENDPOINT}/synthetic/busybox:latest' >/dev/null && docker pull '${REGISTRY_ENDPOINT}/synthetic/busybox:latest' >/dev/null"
+else
+    printf '%-34s skip - set LAB_SYNTHETIC_REGISTRY_PUSH=true to push/pull a tiny image\n' "Registry push/pull smoke"
+fi
+
+section "DNS"
+check "Pi-hole from Debian" dns_query "$PIHOLE_HOST" cloudflare.com
+
+section "Kubernetes Synthetic Checks"
+if kubectl_ok; then
+    check "Cluster DNS temporary pod" kubectl --kubeconfig "$KUBECONFIG_PATH" run synthetic-dns-check --rm -i --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc.cluster.local
+    check "Cluster egress DNS pod" kubectl --kubeconfig "$KUBECONFIG_PATH" run synthetic-egress-dns-check --rm -i --restart=Never --image=busybox:1.36 -- nslookup cloudflare.com
+    check "Ingress from cluster" kubectl --kubeconfig "$KUBECONFIG_PATH" run synthetic-ingress-check --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -ksS --max-time 10 "$PUBLIC_URL"
+else
+    printf 'Kubernetes API unavailable; skipping pod-based checks.\n'
+    warnings=$((warnings + 1))
+fi
+
+section "LoadBalancer"
+check "Traefik LAN HTTP" http_ok "http://${TRAEFIK_LB_IP}/"
+
+section "Summary"
+printf 'failures=%s warnings=%s\n' "$failures" "$warnings"
+
+if [ "$failures" -gt 0 ]; then
+    exit 1
+fi

scripts/validate-homelab

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+run_step() {
+    local label="$1"
+    shift
+
+    printf '\n== %s ==\n' "${label}"
+    "$@"
+}
+
+shell_files() {
+    local path
+
+    git -C "${REPO_ROOT}" ls-files | while IFS= read -r path; do
+        case "${path}" in
+            jeannie | *.sh)
+                printf '%s\n' "${path}"
+                ;;
+            scripts/*)
+                if head -n 1 "${REPO_ROOT}/${path}" | grep -Eq 'sh|bash'; then
+                    printf '%s\n' "${path}"
+                fi
+                ;;
+        esac
+    done
+}
+
+validate_bash_syntax() {
+    local files=()
+    local file
+
+    while IFS= read -r file; do
+        files+=("${REPO_ROOT}/${file}")
+    done < <(shell_files)
+
+    if ((${#files[@]} == 0)); then
+        echo "no shell files found"
+        return 0
+    fi
+    bash -n "${files[@]}"
+}
+
+validate_shellcheck() {
+    local files=()
+    local file
+
+    while IFS= read -r file; do
+        files+=("${REPO_ROOT}/${file}")
+    done < <(shell_files)
+
+    if ! command -v shellcheck >/dev/null 2>&1; then
+        echo "shellcheck not installed; skipping"
+        return 0
+    fi
+    if ((${#files[@]} == 0)); then
+        echo "no shell files found"
+        return 0
+    fi
+    shellcheck "${files[@]}"
+}
+
+validate_yaml() {
+    if command -v ruby >/dev/null 2>&1; then
+        ruby -e 'require "yaml"; (Dir["**/*.yml"] + Dir["**/*.yaml"]).each { |path| next if path.start_with?(".git/"); YAML.load_stream(File.read(path)) }; puts "yaml parse ok"'
+        return 0
+    fi
+    echo "ruby not installed; skipping generic YAML parse"
+}
+
+validate_tofu_fmt() {
+    if ! command -v tofu >/dev/null 2>&1; then
+        echo "tofu not installed; skipping fmt check"
+        return 0
+    fi
+    tofu fmt -check -recursive "${REPO_ROOT}/bootstrap"
+}
+
+validate_security_static() {
+    "${REPO_ROOT}/scripts/validate-tailnet-policy"
+    if command -v trivy >/dev/null 2>&1; then
+        trivy config --quiet "${REPO_ROOT}"
+    else
+        echo "trivy not installed; skipping IaC security static scan"
+    fi
+}
+
+main() {
+    cd "${REPO_ROOT}"
+    run_step "Inventory" "${REPO_ROOT}/scripts/validate-homelab-inventory" "${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}"
+    run_step "Generated docs" "${REPO_ROOT}/scripts/render-docs" --check
+    run_step "Bash syntax" validate_bash_syntax
+    run_step "Shellcheck" validate_shellcheck
+    run_step "YAML parse" validate_yaml
+    run_step "OpenTofu formatting" validate_tofu_fmt
+    run_step "Security static checks" validate_security_static
+}
+
+main "$@"

scripts/workers

@@ -0,0 +1,234 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+KUBECONFIG_PATH="${KUBECONFIG:-${LAB_KUBECONFIG_PATH:-/home/jv/.kube/config}}"
+PIMOX_HOST="${LAB_PIMOX_HOST:-192.168.100.80}"
+PIMOX_USER="${LAB_PIMOX_USER:-jv}"
+PIMOX_KEY="${LAB_PIMOX_SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
+PIMOX_QM_BIN="${LAB_PIMOX_QM_BIN:-/usr/sbin/qm}"
+WORKER_COUNT="${LAB_PIMOX_WORKER_COUNT:-1}"
+WORKER_BASE_VMID="${LAB_PIMOX_WORKER_BASE_VMID:-9010}"
+WORKER_NODE_PREFIX="${LAB_PIMOX_WORKER_NODE_PREFIX:-pimox-worker}"
+WORKER_SKIP_INDEXES="${LAB_PIMOX_SKIP_WORKER_INDEXES:-}"
+
+usage() {
+    cat <<EOF
+Usage: ./jeannie workers <command> [args]
+
+Commands:
+  list                       List desired Pimox workers, VM status, and node status
+  start [index|all]          Start Pimox worker VM(s)
+  stop [index|all]           Gracefully stop Pimox worker VM(s)
+  drain <node>               Drain a Kubernetes worker node
+  uncordon <node>            Mark a Kubernetes worker node schedulable
+  recreate-plan <index>      Print the safe recreate sequence for one Pimox worker
+  rebalance                  Restart evicted/pending pods and show scheduling pressure
+EOF
+}
+
+have() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+is_skipped() {
+    local index="$1"
+    local item
+    local normalized="${WORKER_SKIP_INDEXES//,/ }"
+
+    for item in $normalized; do
+        if [ "$item" = "$index" ]; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+validate_index() {
+    local index="$1"
+
+    if ! [[ "$index" =~ ^[0-9]+$ ]] || [ "$index" -lt 1 ]; then
+        echo "worker index must be a positive integer, got '$index'" >&2
+        exit 1
+    fi
+}
+
+vmid_for_index() {
+    local index="$1"
+    echo $((WORKER_BASE_VMID + index - 1))
+}
+
+node_for_index() {
+    local index="$1"
+    printf '%s-%02d\n' "$WORKER_NODE_PREFIX" "$index"
+}
+
+pimox_ssh() {
+    ssh -i "$PIMOX_KEY" -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${PIMOX_USER}@${PIMOX_HOST}" "$@"
+}
+
+kubectl_cmd() {
+    kubectl --kubeconfig "$KUBECONFIG_PATH" "$@"
+}
+
+target_indexes() {
+    local target="${1:-all}"
+    local index
+
+    if [ "$target" = "all" ]; then
+        for ((index = 1; index <= WORKER_COUNT; index++)); do
+            if ! is_skipped "$index"; then
+                echo "$index"
+            fi
+        done
+    else
+        validate_index "$target"
+        echo "$target"
+    fi
+}
+
+list_workers() {
+    local index
+    local vmid
+    local node
+    local status
+    local node_status
+
+    printf '%-8s %-8s %-24s %-16s %-16s\n' "INDEX" "VMID" "NODE" "VM" "K8S"
+    for ((index = 1; index <= WORKER_COUNT; index++)); do
+        vmid="$(vmid_for_index "$index")"
+        node="$(node_for_index "$index")"
+        if is_skipped "$index"; then
+            printf '%-8s %-8s %-24s %-16s %-16s\n' "$index" "$vmid" "$node" "skipped" "skipped"
+            continue
+        fi
+        status="$(pimox_ssh "sudo '$PIMOX_QM_BIN' status '$vmid'" 2>/dev/null | awk '{print $2}' || echo unknown)"
+        if have kubectl && [ -s "$KUBECONFIG_PATH" ]; then
+            node_status="$(kubectl_cmd get node "$node" --no-headers 2>/dev/null | awk '{print $2}' || echo missing)"
+        else
+            node_status="kubectl-unavailable"
+        fi
+        printf '%-8s %-8s %-24s %-16s %-16s\n' "$index" "$vmid" "$node" "$status" "$node_status"
+    done
+}
+
+start_workers() {
+    local index
+    local vmid
+
+    for index in $(target_indexes "${1:-all}"); do
+        vmid="$(vmid_for_index "$index")"
+        echo "Starting Pimox worker index $index VM $vmid..."
+        pimox_ssh "sudo '$PIMOX_QM_BIN' start '$vmid'"
+    done
+}
+
+stop_workers() {
+    local index
+    local vmid
+
+    for index in $(target_indexes "${1:-all}"); do
+        vmid="$(vmid_for_index "$index")"
+        echo "Gracefully stopping Pimox worker index $index VM $vmid..."
+        pimox_ssh "sudo '$PIMOX_QM_BIN' shutdown '$vmid' --timeout '${LAB_WORKER_SHUTDOWN_TIMEOUT:-90}'" || {
+            echo "Graceful shutdown failed for VM $vmid. Use LAB_CLUSTER_STOP_FORCE=true ./jeannie stop-cluster if a force stop is intended." >&2
+            exit 1
+        }
+    done
+}
+
+drain_node() {
+    local node="${1:-}"
+
+    if [ -z "$node" ]; then
+        echo "workers drain requires a node name." >&2
+        exit 1
+    fi
+    kubectl_cmd drain "$node" --ignore-daemonsets --delete-emptydir-data --timeout="${LAB_WORKER_DRAIN_TIMEOUT:-10m}"
+}
+
+uncordon_node() {
+    local node="${1:-}"
+
+    if [ -z "$node" ]; then
+        echo "workers uncordon requires a node name." >&2
+        exit 1
+    fi
+    kubectl_cmd uncordon "$node"
+}
+
+recreate_plan() {
+    local index="${1:-}"
+    local vmid
+    local node
+
+    validate_index "$index"
+    vmid="$(vmid_for_index "$index")"
+    node="$(node_for_index "$index")"
+    cat <<EOF
+Safe recreate plan for Pimox worker index $index:
+
+1. Snapshot current state:
+   ./jeannie release-snapshot
+
+2. Drain node if it exists:
+   ./jeannie workers drain $node
+
+3. Stop only this VM:
+   ./jeannie workers stop $index
+
+4. Recreate through the normal pipeline so topology and cluster vars stay aligned:
+   LAB_PIMOX_WORKER_COUNT=$WORKER_COUNT LAB_PIMOX_TEMPLATE_REPLACE_EXISTING=false LAB_PIMOX_WORKER_REPLACE_EXISTING=true ./jeannie up
+
+5. Validate:
+   ./jeannie workers list
+   ./jeannie status
+
+Target VMID: $vmid
+Target node: $node
+EOF
+}
+
+rebalance() {
+    if ! have kubectl || [ ! -s "$KUBECONFIG_PATH" ]; then
+        echo "kubectl and kubeconfig are required for workers rebalance." >&2
+        exit 1
+    fi
+    echo "Current node pressure:"
+    kubectl_cmd top nodes 2>/dev/null || kubectl_cmd get nodes -o wide
+    echo
+    echo "Problem pods:"
+    kubectl_cmd get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide || true
+    echo
+    echo "Restarting pending/failed pods is intentionally manual. Use kubectl delete pod for specific pods after checking events."
+}
+
+case "${1:-}" in
+    list)
+        list_workers
+        ;;
+    start)
+        start_workers "${2:-all}"
+        ;;
+    stop)
+        stop_workers "${2:-all}"
+        ;;
+    drain)
+        drain_node "${2:-}"
+        ;;
+    uncordon)
+        uncordon_node "${2:-}"
+        ;;
+    recreate-plan)
+        recreate_plan "${2:-}"
+        ;;
+    rebalance)
+        rebalance
+        ;;
+    -h|--help|help|"")
+        usage
+        ;;
+    *)
+        usage >&2
+        exit 1
+        ;;
+esac

security/README.md

@@ -0,0 +1,87 @@
+# Defensive Security Scans
+
+This directory contains defensive scan configuration for the homelab and public
+website. These checks are for assets owned by this lab only.
+
+Use Jeannie from the Debian homelab host:
+
+```bash
+./jeannie security-prepare
+./jeannie security-scan
+./jeannie security-zap
+./jeannie security-k8s
+./jeannie security-host
+./jeannie security-web
+./jeannie security-logs
+./jeannie security-runtime
+./jeannie security-attack-path
+```
+
+The default `security-scan` runs:
+
+- Trivy filesystem and IaC/config scans against this repo.
+- gitleaks secret scans against this repo.
+- OWASP ZAP baseline passive scans against `security/targets.txt`.
+
+The focused commands are:
+
+- `security-prepare`: validate Docker/report prerequisites and pre-pull scanner
+  images.
+- `security-zap`: OWASP ZAP baseline scan only.
+- `security-k8s`: kube-bench CIS checks against the kubeadm host.
+- `security-host`: Lynis host audit summary for Debian. This command is
+  report-only and prints one highest-risk finding to focus on first.
+- `security-trivy`: Trivy repo and IaC/config scans only.
+- `security-secrets`: gitleaks repo secret scan only.
+- `security-nuclei`: low-rate nuclei HTTP exposure scans only.
+- `security-web`: quick HTTPS, status, and defensive-header checks for owned
+  web targets.
+- `security-logs`: read-only summary of recent Gitea and Traefik security
+  signals.
+- `security-runtime`: Tetragon rollout check and event-log helper.
+- `security-attack-path`: read-only compromise-path drill for a Kubernetes
+  workload.
+
+Reports are written to:
+
+```text
+${HOMELAB_STATE_DIR}/security-reports
+```
+
+The ZAP scan uses baseline/passive mode. Do not run active attack scans against
+public endpoints unless you intentionally schedule a maintenance window and
+understand the traffic it will generate.
+
+`security/targets.txt` is the public target allowlist. Keep it limited to
+services you own.
+
+`security-host` intentionally does not self-fix Lynis findings. Some hardening
+recommendations can break SSH access, Docker networking, kubeadm, or public edge
+routing. When `LAB_BACKSTAGE_BRAIN_ENABLED=true`, Jeannie asks the local
+Ollama/RAG helper to select exactly one finding; otherwise it falls back to the
+first Lynis warning, then the first suggestion. The full Lynis report is still
+saved so you can review the complete output before adding any explicit hardening
+command.
+
+`security-logs` uses recent Docker logs from the Debian Gitea container and
+Traefik pod logs when Kubernetes is reachable. It is intentionally heuristic:
+use it to spot scanner noise, auth failures, suspicious paths, and obvious
+client/server errors before opening the raw logs.
+
+The `apps/security-lab` GitOps app deploys intentionally vulnerable targets for
+practice. They use ClusterIP services only and should be accessed with
+`kubectl port-forward`, not public ingress.
+
+Tetragon is installed by the platform stack as a runtime visibility tool. Use it
+to learn what container process execution looks like after a pod is compromised.
+
+Kyverno hardening policies live under `apps/supply-chain-policy`. The added pod
+hardening rules start in Audit mode so you can review violations before
+promoting any rule to Deny.
+
+`security-attack-path` defaults to the public website workload and can target a
+different workload:
+
+```sh
+./jeannie security-attack-path security-lab deployment/juice-shop
+```

security/targets.txt

@@ -0,0 +1,18 @@
+# Public targets for defensive baseline scans.
+# Keep this list scoped to services you own.
+https://lab2025.duckdns.org/
+https://lab2025.duckdns.org/git/
+https://demos.lab2025.duckdns.org/
+https://heimdall.lab2025.duckdns.org/
+https://argocd.lab2025.duckdns.org/
+https://grafana.lab2025.duckdns.org/
+https://prometheus.lab2025.duckdns.org/
+https://alertmanager.lab2025.duckdns.org/
+https://n8n.lab2025.duckdns.org/
+https://prowlarr.lab2025.duckdns.org/
+https://sonarr.lab2025.duckdns.org/
+https://radarr.lab2025.duckdns.org/
+https://qbittorrent.lab2025.duckdns.org/
+https://kapowarr.lab2025.duckdns.org/
+https://suwayomi.lab2025.duckdns.org/
+https://maintainerr.lab2025.duckdns.org/