Add homelab failure runbooks

README.md

@@ -557,6 +557,12 @@ be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive
 expiry notices, or leave it empty to register without an email. Set
 `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate.
 
+Operational runbooks:
+
+- [Edge failures](docs/runbooks/edge-failures.md)
+- [Gitea failures](docs/runbooks/gitea-failures.md)
+- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md)
+
 ## Adding Apps
 
 Add Kubernetes manifests under `apps/<name>` and register them in

README.md.tmpl

@@ -557,6 +557,12 @@ be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to
 expiry notices, or leave it empty to register without an email. Set
 `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate.
 
+Operational runbooks:
+
+- [Edge failures](docs/runbooks/edge-failures.md)
+- [Gitea failures](docs/runbooks/gitea-failures.md)
+- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md)
+
 ## Adding Apps
 
 Add Kubernetes manifests under `apps/<name>` and register them in

docs/runbooks/cluster-stop-start-failures.md

@@ -0,0 +1,140 @@
+# Cluster Stop/Start Failures Runbook
+
+Use this when `./jeannie stop-cluster` or `./jeannie start-cluster` does not
+complete cleanly, RAM stays high after a stop, or the cluster does not recover
+after a start.
+
+## Intent
+
+`stop-cluster` pauses Kubernetes without destroying kubeadm files, OpenTofu
+state, PV data, or VM disks. It should keep host-level Docker Compose services
+such as Gitea and RPi services separate from the Kubernetes runtime.
+
+## Quick Triage
+
+Run from the Debian host:
+
+```bash
+./jeannie status
+systemctl is-active kubelet containerd docker
+free -h
+sudo crictl ps -a || true
+kubectl get nodes -o wide || true
+```
+
+Check Pimox worker VM state:
+
+```bash
+ssh jv@192.168.100.80 'sudo qm list; sudo qm status 9010; sudo qm status 9011'
+```
+
+## Stop Failures
+
+If Kubernetes containers remain after stop:
+
+```bash
+sudo systemctl stop kubelet
+sudo crictl stop --all || true
+sudo crictl rm --all || true
+```
+
+If RAM is still high, inspect what is actually using memory:
+
+```bash
+ps -eo pid,ppid,comm,%mem,rss,args --sort=-rss | head -40
+sudo crictl ps -a || true
+sudo docker ps
+```
+
+If the memory is from Kubernetes containers, remove stopped CRI containers:
+
+```bash
+sudo crictl rm --all || true
+sudo crictl rmi --prune || true
+```
+
+If the memory is from Docker Compose services, do not stop Gitea or RPi DNS
+unless that service is the incident. Those services intentionally stay online.
+
+If worker VMs did not stop:
+
+```bash
+ssh jv@192.168.100.80 'sudo qm shutdown 9010 --timeout 120 || sudo qm stop 9010'
+ssh jv@192.168.100.80 'sudo qm shutdown 9011 --timeout 120 || sudo qm stop 9011'
+```
+
+## Start Failures
+
+Start the cluster runtime:
+
+```bash
+./jeannie start-cluster
+```
+
+If the API server does not come up:
+
+```bash
+sudo systemctl start containerd
+sudo systemctl start kubelet
+sudo journalctl -u kubelet -n 120 --no-pager
+sudo crictl ps -a | head -50
+```
+
+If nodes stay `NotReady`:
+
+```bash
+kubectl get nodes -o wide
+kubectl describe node debian | sed -n '/Conditions:/,/Addresses:/p'
+kubectl -n kube-system get pods -o wide
+```
+
+Common causes:
+
+- CNI pods are not running.
+- containerd did not start on a worker.
+- Pimox worker VMs are still stopped.
+- Node clocks or DHCP addresses changed after restart.
+
+Start Pimox workers manually if needed:
+
+```bash
+ssh jv@192.168.100.80 'sudo qm start 9010 || true; sudo qm start 9011 || true'
+```
+
+Then wait for nodes:
+
+```bash
+kubectl get nodes -w
+```
+
+## When To Rebuild Instead
+
+Use rebuild only when start cannot recover a coherent cluster:
+
+```bash
+./jeannie rebuild-cluster
+```
+
+Prefer rebuild when:
+
+- kubeadm state is inconsistent.
+- kubelet versions drift from the control plane.
+- worker VM disks/templates are known bad.
+- repeated start attempts leave system pods permanently broken.
+
+## Validation
+
+```bash
+kubectl get nodes -o wide
+kubectl -n kube-system get pods -o wide
+kubectl -n traefik get svc,pods -o wide
+curl -I http://192.168.100.240/
+./jeannie status
+```
+
+Expected:
+
+- Control plane and desired workers are `Ready`.
+- `kube-system` pods are running or completed.
+- Traefik has the MetalLB IP.
+- Host-level Gitea remains up throughout stop/start work.

docs/runbooks/edge-failures.md

@@ -0,0 +1,115 @@
+# Edge Failures Runbook
+
+Use this when `https://lab2025.duckdns.org/`, `/git/`, or a subdomain returns
+`502 Bad Gateway`, times out, or serves the wrong backend.
+
+## Topology
+
+```text
+internet -> OCI jump box nginx -> HAProxy/Varnish/Squid -> homelab backend
+```
+
+Current important backends:
+
+- Website and Kubernetes apps: Traefik MetalLB IP `192.168.100.240:80`
+- Gitea: Debian Tailscale IP `100.85.138.30:3000`
+- OCI edge host: `132.145.170.74`
+
+## Quick Triage
+
+Run from the Debian homelab server:
+
+```bash
+./jeannie status
+curl -I https://lab2025.duckdns.org/
+curl -I https://lab2025.duckdns.org/git/
+ssh ubuntu@132.145.170.74 'sudo docker compose -f /opt/homelab-edge/docker-compose.yml ps'
+```
+
+Check backend reachability from the OCI edge host:
+
+```bash
+ssh ubuntu@132.145.170.74 '
+  curl -I --max-time 5 http://192.168.100.240:80/ || true
+  curl -I --max-time 5 http://100.85.138.30:3000/ || true
+  tailscale status
+'
+```
+
+## Interpret Results
+
+`/git/` works but `/` is `502`:
+
+- Gitea path over Tailscale is healthy.
+- Traefik/MetalLB path is broken or the cluster is stopped.
+- Check Kubernetes services and MetalLB assignment.
+
+`/` works but `/git/` is `502`:
+
+- Edge to Traefik is healthy.
+- Gitea container, Debian Tailscale, or nginx `/git/` proxy path is broken.
+- Use the Gitea runbook.
+
+Both `/` and `/git/` are `502`:
+
+- Edge containers may be unhealthy, DuckDNS may point somewhere unexpected, or
+  the OCI host may have lost Tailscale/private routing.
+
+OCI can reach `100.85.138.30:3000` but public `/git/` is broken:
+
+- Inspect `/opt/homelab-edge/default.conf`.
+- The `/git/` location should proxy to Gitea without forwarding the `/git/`
+  prefix to the backend if Gitea is listening at `/`.
+
+OCI cannot reach `192.168.100.240:80`:
+
+- Check whether the cluster is intentionally stopped.
+- Check Traefik service and MetalLB IP assignment.
+- Check Tailscale subnet route acceptance on the OCI edge host.
+
+## Recovery
+
+Refresh the edge stack from the repo:
+
+```bash
+tofu -chdir=bootstrap/edge apply
+```
+
+Restart edge containers only:
+
+```bash
+ssh ubuntu@132.145.170.74 '
+  cd /opt/homelab-edge
+  sudo docker compose restart
+  sudo docker compose ps
+'
+```
+
+Re-enable accepted Tailscale routes on the edge host:
+
+```bash
+ssh ubuntu@132.145.170.74 'sudo tailscale set --accept-routes=true'
+```
+
+If the cluster was intentionally stopped, either leave website paths down or
+start the cluster:
+
+```bash
+./jeannie start-cluster
+```
+
+## Validation
+
+```bash
+curl -I https://lab2025.duckdns.org/
+curl -I https://lab2025.duckdns.org/git/
+ssh ubuntu@132.145.170.74 '
+  curl -I --max-time 5 http://192.168.100.240:80/ || true
+  curl -I --max-time 5 http://100.85.138.30:3000/ || true
+'
+```
+
+Expected:
+
+- Public requests return `200`, `301`, `302`, or app-specific `404`.
+- Edge backend curls do not fail with connection refused or timeout.

docs/runbooks/gitea-failures.md

@@ -0,0 +1,123 @@
+# Gitea Failures Runbook
+
+Use this when Gitea is down, `/git/` returns `502`, Git push/pull fails, login
+does not work, or the pipeline cannot access the repository.
+
+## Topology
+
+Gitea is intentionally outside Kubernetes:
+
+- Host: Debian `192.168.100.73`
+- Tailscale IP: `100.85.138.30`
+- Install directory: `/data/homelab-gitea`
+- HTTP: `3000`
+- SSH: `32222`
+- Public path: `https://lab2025.duckdns.org/git/`
+
+## Quick Triage
+
+Run on the Debian host:
+
+```bash
+cd ~/my-homelab-configs
+./jeannie status
+cd /data/homelab-gitea
+sudo docker compose ps
+sudo docker compose logs --tail=120 gitea
+curl -I http://127.0.0.1:3000/
+tailscale ip -4
+```
+
+Check the edge path:
+
+```bash
+ssh ubuntu@132.145.170.74 'curl -I --max-time 5 http://100.85.138.30:3000/'
+curl -I https://lab2025.duckdns.org/git/
+```
+
+Check Git SSH from Debian:
+
+```bash
+ssh -T -p 32222 git@192.168.100.73 || true
+git remote -v
+git ls-remote gitea main
+```
+
+## Common Cases
+
+Container is stopped:
+
+```bash
+cd /data/homelab-gitea
+sudo docker compose up -d
+sudo docker compose ps
+```
+
+Docker or containerd is wedged:
+
+```bash
+sudo systemctl restart containerd
+sudo systemctl restart docker
+cd /data/homelab-gitea
+sudo docker compose up -d
+```
+
+Gitea works locally but not from OCI edge:
+
+```bash
+tailscale ip -4
+ssh ubuntu@132.145.170.74 'tailscale status; curl -I --max-time 5 http://100.85.138.30:3000/'
+```
+
+If the Debian Tailscale IP changed, update the edge variable and reapply:
+
+```bash
+tofu -chdir=bootstrap/edge apply
+```
+
+Git HTTPS prompts or fails authentication:
+
+- Prefer SSH for writes from the homelab.
+- Keep the Debian working copy remote pointed at the LAN SSH endpoint:
+
+```bash
+git remote set-url gitea ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git
+git branch --set-upstream-to=gitea/main main
+```
+
+Login password is lost:
+
+```bash
+cd /data/homelab-gitea
+sudo docker exec -it homelab-gitea gitea admin user list --config /data/gitea/conf/app.ini
+sudo docker exec -it homelab-gitea gitea admin user change-password \
+  --config /data/gitea/conf/app.ini \
+  --username jv \
+  --password '<new-password>'
+```
+
+## Safe Redeploy
+
+Redeploying Gitea should not delete data because the Compose stack uses the
+persistent data directory under `/data/homelab-gitea/data`.
+
+```bash
+cd ~/my-homelab-configs
+./jeannie deploy-gitea
+./jeannie bootstrap-gitea-repo
+```
+
+## Validation
+
+```bash
+curl -I http://127.0.0.1:3000/
+curl -I https://lab2025.duckdns.org/git/
+git ls-remote gitea main
+sudo docker compose -f /data/homelab-gitea/docker-compose.yml ps
+```
+
+Expected:
+
+- Local HTTP returns a web response.
+- Public `/git/` loads through edge.
+- `git ls-remote` returns the `main` ref.