Allow Debian Docker root on NVMe root filesystem

README.md

@@ -108,7 +108,7 @@ On the Debian host:
   minor, currently `v1.36`
 - SSH access to worker nodes
 - SSH access to the OCI edge host
-- enough persistent storage for `/data/openebs/local` and `/data/docker`
+- enough persistent storage for `/data/openebs/local` and Docker's data root
 
 After a clean Debian reinstall, bootstrap the host package set and desktop
 configuration first:
@@ -230,6 +230,13 @@ or host storage/networking:
 ./jeannie preflight
 ```
 
+If the Debian Docker root check fails after a reinstall, repair it before
+running the full pipeline:
+
+```bash
+./jeannie fix-debian-docker-root
+```
+
 ## Validation
 
 Useful checks after a rebuild:
@@ -665,11 +672,10 @@ destroy/create cycles. Those critical volumes are declared explicitly as
 retained local PVs so a rebuilt cluster binds back to the same host paths
 instead of creating fresh directories.
 
-For the current lab, the HP Debian laptop's NVMe is managed by LVM volume group
+For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
-`data-vg` and mounted under `/data`. OpenEBS retained hostpath data lives under
+standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
-`/data/openebs/local`, and Docker data should live under `/data/docker` where
+data still lives under `/data/openebs/local`, and larger service data such as
-the host is configured that way. This keeps retained PVs, container layers,
+Gitea, Ollama models, and app volumes should stay under `/data`.
-Buildx state, and image caches from filling `/`.
 
 ## Gitea
 
@@ -678,9 +684,8 @@ always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
 Kubernetes workload. This keeps Git available when the Kubernetes cluster is
 destroyed and rebuilt.
 
-The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. The
+The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
-Docker root is also expected to live on the `/data` storage path rather than the
+Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.
-root filesystem.
 
 Public source browsing stays available through
 `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users

README.md.tmpl

@@ -108,7 +108,7 @@ On the Debian host:
   minor, currently `v1.36`
 - SSH access to worker nodes
 - SSH access to the OCI edge host
-- enough persistent storage for `/data/openebs/local` and `/data/docker`
+- enough persistent storage for `/data/openebs/local` and Docker's data root
 
 After a clean Debian reinstall, bootstrap the host package set and desktop
 configuration first:
@@ -230,6 +230,13 @@ or host storage/networking:
 ./{{ main_script }} preflight
 ```
 
+If the Debian Docker root check fails after a reinstall, repair it before
+running the full pipeline:
+
+```bash
+./{{ main_script }} fix-debian-docker-root
+```
+
 ## Validation
 
 Useful checks after a rebuild:
@@ -665,11 +672,10 @@ destroy/create cycles. Those critical volumes are declared explicitly as
 retained local PVs so a rebuilt cluster binds back to the same host paths
 instead of creating fresh directories.
 
-For the current lab, the HP Debian laptop's NVMe is managed by LVM volume group
+For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
-`data-vg` and mounted under `/data`. OpenEBS retained hostpath data lives under
+standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
-`/data/openebs/local`, and Docker data should live under `/data/docker` where
+data still lives under `/data/openebs/local`, and larger service data such as
-the host is configured that way. This keeps retained PVs, container layers,
+Gitea, Ollama models, and app volumes should stay under `/data`.
-Buildx state, and image caches from filling `/`.
 
 ## Gitea
 
@@ -678,9 +684,8 @@ always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
 Kubernetes workload. This keeps Git available when the Kubernetes cluster is
 destroyed and rebuilt.
 
-The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. The
+The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
-Docker root is also expected to live on the `/data` storage path rather than the
+Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.
-root filesystem.
 
 Public source browsing stays available through
 `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users

apps/demos-static/public/homelab-catalog/index.html

@@ -25,7 +25,7 @@
           <h2>debian</h2>
           <p><strong>Address:</strong> 192.168.100.73</p>
           <p><strong>Tailscale:</strong> 100.85.138.30</p>
-          <p><strong>Storage:</strong> /data/docker</p>
+          <p><strong>Storage:</strong> /var/lib/docker</p>
         </article>
         <article class="catalog-card service-host">
           <span>dns-edge-worker</span>

bootstrap/host/group_vars/debian_pc.yml

@@ -16,7 +16,6 @@ debian_pc_ollama_igpu_enable: "false"
 
 debian_pc_persistent_directories:
   - /data/openebs/local/registry
-  - /data/docker
   - /data/ollama/models
   - "/home/{{ debian_pc_user }}/git-server"
   - "/home/{{ debian_pc_user }}/backups/gitea"

docs/service-catalog.md

@@ -9,7 +9,7 @@ Update `homelab.yml`, then run `scripts/render-service-catalog`.
 
 | Host | Role | Address | Tailscale | Storage |
 | --- | --- | --- | --- | --- |
-| debian | control-plane | 192.168.100.73 | 100.85.138.30 | /data/docker |
+| debian | control-plane | 192.168.100.73 | 100.85.138.30 | /var/lib/docker |
 | rpi4 | dns-edge-worker | 192.168.100.89 | 100.77.80.72 | /nvme-storage/docker |
 | opi5_pimox | pimox | 192.168.100.80 |  | opi5_ssd |
 | oci_edge | public-edge | 132.145.170.74 | 100.118.255.19 |  |

homelab.yml

@@ -39,7 +39,7 @@ hosts:
     user: jv
     lan_ip: 192.168.100.73
     tailscale_ip: 100.85.138.30
-    docker_root: /data/docker
+    docker_root: /var/lib/docker
     data_root: /data
     kubeconfig: /home/jv/.kube/config
   rpi4:

infra/rpi-services/seed-uptime-kuma.js

@@ -2,9 +2,59 @@
 "use strict";
 
 const fs = require("fs");
+const path = require("path");
+
+function findModuleDirectory(root, moduleName, depth = 0) {
+  if (depth > 5) {
+    return "";
+  }
+  let entries;
+  try {
+    entries = fs.readdirSync(root, { withFileTypes: true });
+  } catch (_error) {
+    return "";
+  }
+
+  for (const entry of entries) {
+    if (!entry.isDirectory()) {
+      continue;
+    }
+    const fullPath = path.join(root, entry.name);
+    if (entry.name === moduleName && fs.existsSync(path.join(fullPath, "package.json"))) {
+      return fullPath;
+    }
+  }
+
+  for (const entry of entries) {
+    if (!entry.isDirectory()) {
+      continue;
+    }
+    const fullPath = path.join(root, entry.name);
+    const found = findModuleDirectory(fullPath, moduleName, depth + 1);
+    if (found) {
+      return found;
+    }
+  }
+
+  return "";
+}
 
 function loadDatabaseModule() {
-  for (const moduleName of ["better-sqlite3", "/app/node_modules/better-sqlite3"]) {
+  const candidates = [
+    "better-sqlite3",
+    "/app/node_modules/better-sqlite3",
+    "/app/server/node_modules/better-sqlite3",
+    "/app/src/node_modules/better-sqlite3"
+  ];
+
+  for (const root of ["/app/node_modules", "/app/server/node_modules", "/app/src/node_modules"]) {
+    const found = findModuleDirectory(root, "better-sqlite3");
+    if (found) {
+      candidates.push(found);
+    }
+  }
+
+  for (const moduleName of candidates) {
     try {
       return require(moduleName);
     } catch (_error) {

jeannie

@@ -1067,7 +1067,7 @@ check_gitea_reachable() {
 }
 
 check_debian_docker_root() {
-    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/data/docker}"
+    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}"
     local actual_root
 
     if ! command -v docker >/dev/null 2>&1; then
@@ -1081,6 +1081,60 @@ check_debian_docker_root() {
     fi
 }
 
+fix_debian_docker_root() {
+    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}"
+    local daemon_file="/etc/docker/daemon.json"
+    local current_root=""
+
+    require_debian_server "fix-debian-docker-root"
+
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "docker command was not found on Debian host" >&2
+        exit 1
+    fi
+
+    current_root="$(sudo docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)"
+    if [[ "${current_root}" == "${expected_root}" ]]; then
+        echo "Debian Docker root is already ${expected_root}."
+        return 0
+    fi
+
+    echo "Moving Debian Docker root from ${current_root:-unknown} to ${expected_root}..."
+    sudo mkdir -p "${expected_root}" /etc/docker
+
+    echo "Stopping Docker services..."
+    sudo systemctl stop docker 2>/dev/null || true
+    sudo systemctl stop containerd 2>/dev/null || true
+
+    if [[ -n "${current_root}" && -d "${current_root}" && "${current_root}" != "${expected_root}" ]]; then
+        echo "Copying existing Docker data to ${expected_root}..."
+        sudo rsync -aHAX --numeric-ids "${current_root}/" "${expected_root}/"
+    fi
+
+    sudo python3 - "${daemon_file}" "${expected_root}" <<'PY'
+import json
+import os
+import sys
+
+path, expected_root = sys.argv[1:3]
+document = {}
+if os.path.exists(path) and os.path.getsize(path) > 0:
+    with open(path, encoding="utf-8") as handle:
+        document = json.load(handle)
+document["data-root"] = expected_root
+with open(path, "w", encoding="utf-8") as handle:
+    json.dump(document, handle, indent=2, sort_keys=True)
+    handle.write("\n")
+PY
+
+    echo "Starting Docker services..."
+    sudo systemctl start containerd
+    sudo systemctl start docker
+
+    check_debian_docker_root
+    echo "Debian Docker root is now ${expected_root}."
+}
+
 check_debian_tailscale_ip() {
     local expected_ip="${LAB_DEBIAN_TAILSCALE_IP:-}"
 
@@ -1174,7 +1228,7 @@ homelab_preflight() {
     echo "Running homelab preflight checks from ${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}..."
 
     preflight_check "Gitea reachable at ${LAB_GITEA_LOCAL_URL:-http://${LAB_GITEA_HOST:-192.168.100.73}:${LAB_GITEA_HTTP_PORT:-3000}/}" check_gitea_reachable || failures=$((failures + 1))
-    preflight_check "Debian Docker root is ${LAB_DEBIAN_DOCKER_ROOT:-/data/docker}" check_debian_docker_root || failures=$((failures + 1))
+    preflight_check "Debian Docker root is ${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}" check_debian_docker_root || failures=$((failures + 1))
     preflight_check "Debian Tailscale IP ${LAB_DEBIAN_TAILSCALE_IP:-unset}" check_debian_tailscale_ip || failures=$((failures + 1))
     preflight_check "RPi Docker root is NVMe or fallback" check_rpi_docker_root_state || failures=$((failures + 1))
     preflight_warn "RPi Tailscale IP ${LAB_RPI_TAILSCALE_IP:-unset}" check_rpi_tailscale_ip
@@ -4286,7 +4340,7 @@ status_report() {
     printf 'date:     %s\n' "$(date -Is)"
     printf 'uptime:   %s\n' "$(uptime -p 2>/dev/null || uptime)"
     status_run "memory" free -h
-    status_run "disk" df -h / /data /data/docker /data/openebs/local
+    status_run "disk" df -h / /data /data/openebs/local
 
     status_section "Systemd"
     status_systemd_units ssh docker containerd kubelet tailscaled homelab-gitea-runner
@@ -4921,6 +4975,9 @@ case "${1:-}" in
     preflight)
         homelab_preflight
         ;;
+    fix-debian-docker-root)
+        fix_debian_docker_root
+        ;;
     secrets-init)
         secrets_init
         ;;
@@ -4943,7 +5000,7 @@ case "${1:-}" in
         nuke
         ;;
     *)
-        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|ollama-setup|deploy-gitea|rpi-services|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|doctor-edge|doctor-gitea|doctor-rpi|doctor-cluster|preflight|secrets-init|secrets-check|tailnet-policy-check|ai-index|ai-check|openwrt|nuke}"
+        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|ollama-setup|deploy-gitea|rpi-services|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|doctor-edge|doctor-gitea|doctor-rpi|doctor-cluster|preflight|fix-debian-docker-root|secrets-init|secrets-check|tailnet-policy-check|ai-index|ai-check|openwrt|nuke}"
         exit 1
         ;;
 esac