Allow Debian Docker root on NVMe root filesystem

Add Debian Docker root repair command
2026-06-29 11:14:59 -06:00 · 2026-06-29 11:13:29 -06:00
8 changed files with 143 additions and 27 deletions
--- a/README.md
+++ b/README.md
@ -108,7 +108,7 @@ On the Debian host:
  minor, currently `v1.36`
 - SSH access to worker nodes
 - SSH access to the OCI edge host
- enough persistent storage for `/data/openebs/local` and `/data/docker`
+- enough persistent storage for `/data/openebs/local` and Docker's data root

 After a clean Debian reinstall, bootstrap the host package set and desktop
 configuration first:
@ -230,6 +230,13 @@ or host storage/networking:
 ./jeannie preflight
 ```

+If the Debian Docker root check fails after a reinstall, repair it before
+running the full pipeline:
+
+```bash
+./jeannie fix-debian-docker-root
+```
+
 ## Validation

 Useful checks after a rebuild:
@ -665,11 +672,10 @@ destroy/create cycles. Those critical volumes are declared explicitly as
 retained local PVs so a rebuilt cluster binds back to the same host paths
 instead of creating fresh directories.

-For the current lab, the HP Debian laptop's NVMe is managed by LVM volume group
-`data-vg` and mounted under `/data`. OpenEBS retained hostpath data lives under
-`/data/openebs/local`, and Docker data should live under `/data/docker` where
-the host is configured that way. This keeps retained PVs, container layers,
-Buildx state, and image caches from filling `/`.
+For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
+standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
+data still lives under `/data/openebs/local`, and larger service data such as
+Gitea, Ollama models, and app volumes should stay under `/data`.

 ## Gitea

@ -678,9 +684,8 @@ always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
 Kubernetes workload. This keeps Git available when the Kubernetes cluster is
 destroyed and rebuilt.

-The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. The
-Docker root is also expected to live on the `/data` storage path rather than the
-root filesystem.
+The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
+Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.

 Public source browsing stays available through
 `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
--- a/README.md.tmpl
+++ b/README.md.tmpl
@ -108,7 +108,7 @@ On the Debian host:
  minor, currently `v1.36`
 - SSH access to worker nodes
 - SSH access to the OCI edge host
- enough persistent storage for `/data/openebs/local` and `/data/docker`
+- enough persistent storage for `/data/openebs/local` and Docker's data root

 After a clean Debian reinstall, bootstrap the host package set and desktop
 configuration first:
@ -230,6 +230,13 @@ or host storage/networking:
 ./{{ main_script }} preflight
 ```

+If the Debian Docker root check fails after a reinstall, repair it before
+running the full pipeline:
+
+```bash
+./{{ main_script }} fix-debian-docker-root
+```
+
 ## Validation

 Useful checks after a rebuild:
@ -665,11 +672,10 @@ destroy/create cycles. Those critical volumes are declared explicitly as
 retained local PVs so a rebuilt cluster binds back to the same host paths
 instead of creating fresh directories.

-For the current lab, the HP Debian laptop's NVMe is managed by LVM volume group
-`data-vg` and mounted under `/data`. OpenEBS retained hostpath data lives under
-`/data/openebs/local`, and Docker data should live under `/data/docker` where
-the host is configured that way. This keeps retained PVs, container layers,
-Buildx state, and image caches from filling `/`.
+For the current lab, the HP Debian laptop's root filesystem is on NVMe, so the
+standard Docker root `/var/lib/docker` is acceptable. OpenEBS retained hostpath
+data still lives under `/data/openebs/local`, and larger service data such as
+Gitea, Ollama models, and app volumes should stay under `/data`.

 ## Gitea

@ -678,9 +684,8 @@ always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
 Kubernetes workload. This keeps Git available when the Kubernetes cluster is
 destroyed and rebuilt.

-The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe. The
-Docker root is also expected to live on the `/data` storage path rather than the
-root filesystem.
+The default data path is `/data/homelab-gitea/data` on the HP laptop NVMe.
+Docker may use the standard `/var/lib/docker` root because `/` is also on NVMe.

 Public source browsing stays available through
 `https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
--- a/apps/demos-static/public/homelab-catalog/index.html
+++ b/apps/demos-static/public/homelab-catalog/index.html
@ -25,7 +25,7 @@
          <h2>debian</h2>
          <p><strong>Address:</strong> 192.168.100.73</p>
          <p><strong>Tailscale:</strong> 100.85.138.30</p>
-          <p><strong>Storage:</strong> /data/docker</p>
+          <p><strong>Storage:</strong> /var/lib/docker</p>
        </article>
        <article class="catalog-card service-host">
          <span>dns-edge-worker</span>
--- a/bootstrap/host/group_vars/debian_pc.yml
+++ b/bootstrap/host/group_vars/debian_pc.yml
@ -16,7 +16,6 @@ debian_pc_ollama_igpu_enable: "false"

 debian_pc_persistent_directories:
  - /data/openebs/local/registry
-  - /data/docker
  - /data/ollama/models
  - "/home/{{ debian_pc_user }}/git-server"
  - "/home/{{ debian_pc_user }}/backups/gitea"
--- a/docs/service-catalog.md
+++ b/docs/service-catalog.md
@ -9,7 +9,7 @@ Update `homelab.yml`, then run `scripts/render-service-catalog`.

 | Host | Role | Address | Tailscale | Storage |
 | --- | --- | --- | --- | --- |
-| debian | control-plane | 192.168.100.73 | 100.85.138.30 | /data/docker |
+| debian | control-plane | 192.168.100.73 | 100.85.138.30 | /var/lib/docker |
 | rpi4 | dns-edge-worker | 192.168.100.89 | 100.77.80.72 | /nvme-storage/docker |
 | opi5_pimox | pimox | 192.168.100.80 |  | opi5_ssd |
 | oci_edge | public-edge | 132.145.170.74 | 100.118.255.19 |  |
--- a/homelab.yml
+++ b/homelab.yml
@ -39,7 +39,7 @@ hosts:
    user: jv
    lan_ip: 192.168.100.73
    tailscale_ip: 100.85.138.30
-    docker_root: /data/docker
+    docker_root: /var/lib/docker
    data_root: /data
    kubeconfig: /home/jv/.kube/config
  rpi4:
--- a/infra/rpi-services/seed-uptime-kuma.js
+++ b/infra/rpi-services/seed-uptime-kuma.js
@ -2,9 +2,59 @@
 "use strict";

 const fs = require("fs");
+const path = require("path");
+
+function findModuleDirectory(root, moduleName, depth = 0) {
+  if (depth > 5) {
+    return "";
+  }
+  let entries;
+  try {
+    entries = fs.readdirSync(root, { withFileTypes: true });
+  } catch (_error) {
+    return "";
+  }
+
+  for (const entry of entries) {
+    if (!entry.isDirectory()) {
+      continue;
+    }
+    const fullPath = path.join(root, entry.name);
+    if (entry.name === moduleName && fs.existsSync(path.join(fullPath, "package.json"))) {
+      return fullPath;
+    }
+  }
+
+  for (const entry of entries) {
+    if (!entry.isDirectory()) {
+      continue;
+    }
+    const fullPath = path.join(root, entry.name);
+    const found = findModuleDirectory(fullPath, moduleName, depth + 1);
+    if (found) {
+      return found;
+    }
+  }
+
+  return "";
+}

 function loadDatabaseModule() {
-  for (const moduleName of ["better-sqlite3", "/app/node_modules/better-sqlite3"]) {
+  const candidates = [
+    "better-sqlite3",
+    "/app/node_modules/better-sqlite3",
+    "/app/server/node_modules/better-sqlite3",
+    "/app/src/node_modules/better-sqlite3"
+  ];
+
+  for (const root of ["/app/node_modules", "/app/server/node_modules", "/app/src/node_modules"]) {
+    const found = findModuleDirectory(root, "better-sqlite3");
+    if (found) {
+      candidates.push(found);
+    }
+  }
+
+  for (const moduleName of candidates) {
    try {
      return require(moduleName);
    } catch (_error) {
--- a/65
+++ b/65
@ -1067,7 +1067,7 @@ check_gitea_reachable() {
 }

 check_debian_docker_root() {
-    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/data/docker}"
+    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}"
    local actual_root

    if ! command -v docker >/dev/null 2>&1; then
@ -1081,6 +1081,60 @@ check_debian_docker_root() {
    fi
 }

+fix_debian_docker_root() {
+    local expected_root="${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}"
+    local daemon_file="/etc/docker/daemon.json"
+    local current_root=""
+
+    require_debian_server "fix-debian-docker-root"
+
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "docker command was not found on Debian host" >&2
+        exit 1
+    fi
+
+    current_root="$(sudo docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)"
+    if [[ "${current_root}" == "${expected_root}" ]]; then
+        echo "Debian Docker root is already ${expected_root}."
+        return 0
+    fi
+
+    echo "Moving Debian Docker root from ${current_root:-unknown} to ${expected_root}..."
+    sudo mkdir -p "${expected_root}" /etc/docker
+
+    echo "Stopping Docker services..."
+    sudo systemctl stop docker 2>/dev/null || true
+    sudo systemctl stop containerd 2>/dev/null || true
+
+    if [[ -n "${current_root}" && -d "${current_root}" && "${current_root}" != "${expected_root}" ]]; then
+        echo "Copying existing Docker data to ${expected_root}..."
+        sudo rsync -aHAX --numeric-ids "${current_root}/" "${expected_root}/"
+    fi
+
+    sudo python3 - "${daemon_file}" "${expected_root}" <<'PY'
+import json
+import os
+import sys
+
+path, expected_root = sys.argv[1:3]
+document = {}
+if os.path.exists(path) and os.path.getsize(path) > 0:
+    with open(path, encoding="utf-8") as handle:
+        document = json.load(handle)
+document["data-root"] = expected_root
+with open(path, "w", encoding="utf-8") as handle:
+    json.dump(document, handle, indent=2, sort_keys=True)
+    handle.write("\n")
+PY
+
+    echo "Starting Docker services..."
+    sudo systemctl start containerd
+    sudo systemctl start docker
+
+    check_debian_docker_root
+    echo "Debian Docker root is now ${expected_root}."
+}
+
 check_debian_tailscale_ip() {
    local expected_ip="${LAB_DEBIAN_TAILSCALE_IP:-}"

@ -1174,7 +1228,7 @@ homelab_preflight() {
    echo "Running homelab preflight checks from ${HOMELAB_INVENTORY_FILE:-${REPO_ROOT}/homelab.yml}..."

    preflight_check "Gitea reachable at ${LAB_GITEA_LOCAL_URL:-http://${LAB_GITEA_HOST:-192.168.100.73}:${LAB_GITEA_HTTP_PORT:-3000}/}" check_gitea_reachable || failures=$((failures + 1))
-    preflight_check "Debian Docker root is ${LAB_DEBIAN_DOCKER_ROOT:-/data/docker}" check_debian_docker_root || failures=$((failures + 1))
+    preflight_check "Debian Docker root is ${LAB_DEBIAN_DOCKER_ROOT:-/var/lib/docker}" check_debian_docker_root || failures=$((failures + 1))
    preflight_check "Debian Tailscale IP ${LAB_DEBIAN_TAILSCALE_IP:-unset}" check_debian_tailscale_ip || failures=$((failures + 1))
    preflight_check "RPi Docker root is NVMe or fallback" check_rpi_docker_root_state || failures=$((failures + 1))
    preflight_warn "RPi Tailscale IP ${LAB_RPI_TAILSCALE_IP:-unset}" check_rpi_tailscale_ip
@ -4286,7 +4340,7 @@ status_report() {
    printf 'date:     %s\n' "$(date -Is)"
    printf 'uptime:   %s\n' "$(uptime -p 2>/dev/null || uptime)"
    status_run "memory" free -h
-    status_run "disk" df -h / /data /data/docker /data/openebs/local
+    status_run "disk" df -h / /data /data/openebs/local

    status_section "Systemd"
    status_systemd_units ssh docker containerd kubelet tailscaled homelab-gitea-runner
@ -4921,6 +4975,9 @@ case "${1:-}" in
    preflight)
        homelab_preflight
        ;;
+    fix-debian-docker-root)
+        fix_debian_docker_root
+        ;;
    secrets-init)
        secrets_init
        ;;
@ -4943,7 +5000,7 @@ case "${1:-}" in
        nuke
        ;;
    *)
-        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|ollama-setup|deploy-gitea|rpi-services|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|doctor-edge|doctor-gitea|doctor-rpi|doctor-cluster|preflight|secrets-init|secrets-check|tailnet-policy-check|ai-index|ai-check|openwrt|nuke}"
+        echo "Usage: $0 {up|rebuild-cluster|stop-cluster|start-cluster|status|apps|website-translation-model|website-ollama-listen|ollama-setup|deploy-gitea|rpi-services|bootstrap-gitea-repo|backup-gitea|drill-gitea-restore|install-gitea-runner|move-prometheus-stack-workers|doctor-versions|doctor-edge|doctor-gitea|doctor-rpi|doctor-cluster|preflight|fix-debian-docker-root|secrets-init|secrets-check|tailnet-policy-check|ai-index|ai-check|openwrt|nuke}"
        exit 1
        ;;
 esac
Author	SHA1	Message	Date
juvdiaz	3c24f05ff6	Allow Debian Docker root on NVMe root filesystem	2026-06-29 11:14:59 -06:00
juvdiaz	b49df4f2eb	Add Debian Docker root repair command	2026-06-29 11:13:29 -06:00