From 6ee2e2cb46ff3a71bd621a2ebaf0fee7bb51b12d Mon Sep 17 00:00:00 2001 From: juvdiaz Date: Sat, 27 Jun 2026 16:23:09 -0600 Subject: [PATCH] Add homelab failure runbooks --- README.md | 6 + README.md.tmpl | 6 + docs/runbooks/cluster-stop-start-failures.md | 140 +++++++++++++++++++ docs/runbooks/edge-failures.md | 115 +++++++++++++++ docs/runbooks/gitea-failures.md | 123 ++++++++++++++++ 5 files changed, 390 insertions(+) create mode 100644 docs/runbooks/cluster-stop-start-failures.md create mode 100644 docs/runbooks/edge-failures.md create mode 100644 docs/runbooks/gitea-failures.md diff --git a/README.md b/README.md index a214874..687cbd8 100644 --- a/README.md +++ b/README.md @@ -557,6 +557,12 @@ be open before `./jeannie up` runs. Set `TF_VAR_letsencrypt_email` to receive expiry notices, or leave it empty to register without an email. Set `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + ## Adding Apps Add Kubernetes manifests under `apps/` and register them in diff --git a/README.md.tmpl b/README.md.tmpl index cfb2a74..c1475df 100644 --- a/README.md.tmpl +++ b/README.md.tmpl @@ -557,6 +557,12 @@ be open before `./{{ main_script }} up` runs. Set `TF_VAR_letsencrypt_email` to expiry notices, or leave it empty to register without an email. Set `TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate. +Operational runbooks: + +- [Edge failures](docs/runbooks/edge-failures.md) +- [Gitea failures](docs/runbooks/gitea-failures.md) +- [Cluster stop/start failures](docs/runbooks/cluster-stop-start-failures.md) + ## Adding Apps Add Kubernetes manifests under `apps/` and register them in diff --git a/docs/runbooks/cluster-stop-start-failures.md b/docs/runbooks/cluster-stop-start-failures.md new file mode 100644 index 0000000..db8d2a3 --- /dev/null +++ b/docs/runbooks/cluster-stop-start-failures.md @@ -0,0 +1,140 @@ +# Cluster Stop/Start Failures Runbook + +Use this when `./jeannie stop-cluster` or `./jeannie start-cluster` does not +complete cleanly, RAM stays high after a stop, or the cluster does not recover +after a start. + +## Intent + +`stop-cluster` pauses Kubernetes without destroying kubeadm files, OpenTofu +state, PV data, or VM disks. It should keep host-level Docker Compose services +such as Gitea and RPi services separate from the Kubernetes runtime. + +## Quick Triage + +Run from the Debian host: + +```bash +./jeannie status +systemctl is-active kubelet containerd docker +free -h +sudo crictl ps -a || true +kubectl get nodes -o wide || true +``` + +Check Pimox worker VM state: + +```bash +ssh jv@192.168.100.80 'sudo qm list; sudo qm status 9010; sudo qm status 9011' +``` + +## Stop Failures + +If Kubernetes containers remain after stop: + +```bash +sudo systemctl stop kubelet +sudo crictl stop --all || true +sudo crictl rm --all || true +``` + +If RAM is still high, inspect what is actually using memory: + +```bash +ps -eo pid,ppid,comm,%mem,rss,args --sort=-rss | head -40 +sudo crictl ps -a || true +sudo docker ps +``` + +If the memory is from Kubernetes containers, remove stopped CRI containers: + +```bash +sudo crictl rm --all || true +sudo crictl rmi --prune || true +``` + +If the memory is from Docker Compose services, do not stop Gitea or RPi DNS +unless that service is the incident. Those services intentionally stay online. + +If worker VMs did not stop: + +```bash +ssh jv@192.168.100.80 'sudo qm shutdown 9010 --timeout 120 || sudo qm stop 9010' +ssh jv@192.168.100.80 'sudo qm shutdown 9011 --timeout 120 || sudo qm stop 9011' +``` + +## Start Failures + +Start the cluster runtime: + +```bash +./jeannie start-cluster +``` + +If the API server does not come up: + +```bash +sudo systemctl start containerd +sudo systemctl start kubelet +sudo journalctl -u kubelet -n 120 --no-pager +sudo crictl ps -a | head -50 +``` + +If nodes stay `NotReady`: + +```bash +kubectl get nodes -o wide +kubectl describe node debian | sed -n '/Conditions:/,/Addresses:/p' +kubectl -n kube-system get pods -o wide +``` + +Common causes: + +- CNI pods are not running. +- containerd did not start on a worker. +- Pimox worker VMs are still stopped. +- Node clocks or DHCP addresses changed after restart. + +Start Pimox workers manually if needed: + +```bash +ssh jv@192.168.100.80 'sudo qm start 9010 || true; sudo qm start 9011 || true' +``` + +Then wait for nodes: + +```bash +kubectl get nodes -w +``` + +## When To Rebuild Instead + +Use rebuild only when start cannot recover a coherent cluster: + +```bash +./jeannie rebuild-cluster +``` + +Prefer rebuild when: + +- kubeadm state is inconsistent. +- kubelet versions drift from the control plane. +- worker VM disks/templates are known bad. +- repeated start attempts leave system pods permanently broken. + +## Validation + +```bash +kubectl get nodes -o wide +kubectl -n kube-system get pods -o wide +kubectl -n traefik get svc,pods -o wide +curl -I http://192.168.100.240/ +./jeannie status +``` + +Expected: + +- Control plane and desired workers are `Ready`. +- `kube-system` pods are running or completed. +- Traefik has the MetalLB IP. +- Host-level Gitea remains up throughout stop/start work. diff --git a/docs/runbooks/edge-failures.md b/docs/runbooks/edge-failures.md new file mode 100644 index 0000000..3011b20 --- /dev/null +++ b/docs/runbooks/edge-failures.md @@ -0,0 +1,115 @@ +# Edge Failures Runbook + +Use this when `https://lab2025.duckdns.org/`, `/git/`, or a subdomain returns +`502 Bad Gateway`, times out, or serves the wrong backend. + +## Topology + +```text +internet -> OCI jump box nginx -> HAProxy/Varnish/Squid -> homelab backend +``` + +Current important backends: + +- Website and Kubernetes apps: Traefik MetalLB IP `192.168.100.240:80` +- Gitea: Debian Tailscale IP `100.85.138.30:3000` +- OCI edge host: `132.145.170.74` + +## Quick Triage + +Run from the Debian homelab server: + +```bash +./jeannie status +curl -I https://lab2025.duckdns.org/ +curl -I https://lab2025.duckdns.org/git/ +ssh ubuntu@132.145.170.74 'sudo docker compose -f /opt/homelab-edge/docker-compose.yml ps' +``` + +Check backend reachability from the OCI edge host: + +```bash +ssh ubuntu@132.145.170.74 ' + curl -I --max-time 5 http://192.168.100.240:80/ || true + curl -I --max-time 5 http://100.85.138.30:3000/ || true + tailscale status +' +``` + +## Interpret Results + +`/git/` works but `/` is `502`: + +- Gitea path over Tailscale is healthy. +- Traefik/MetalLB path is broken or the cluster is stopped. +- Check Kubernetes services and MetalLB assignment. + +`/` works but `/git/` is `502`: + +- Edge to Traefik is healthy. +- Gitea container, Debian Tailscale, or nginx `/git/` proxy path is broken. +- Use the Gitea runbook. + +Both `/` and `/git/` are `502`: + +- Edge containers may be unhealthy, DuckDNS may point somewhere unexpected, or + the OCI host may have lost Tailscale/private routing. + +OCI can reach `100.85.138.30:3000` but public `/git/` is broken: + +- Inspect `/opt/homelab-edge/default.conf`. +- The `/git/` location should proxy to Gitea without forwarding the `/git/` + prefix to the backend if Gitea is listening at `/`. + +OCI cannot reach `192.168.100.240:80`: + +- Check whether the cluster is intentionally stopped. +- Check Traefik service and MetalLB IP assignment. +- Check Tailscale subnet route acceptance on the OCI edge host. + +## Recovery + +Refresh the edge stack from the repo: + +```bash +tofu -chdir=bootstrap/edge apply +``` + +Restart edge containers only: + +```bash +ssh ubuntu@132.145.170.74 ' + cd /opt/homelab-edge + sudo docker compose restart + sudo docker compose ps +' +``` + +Re-enable accepted Tailscale routes on the edge host: + +```bash +ssh ubuntu@132.145.170.74 'sudo tailscale set --accept-routes=true' +``` + +If the cluster was intentionally stopped, either leave website paths down or +start the cluster: + +```bash +./jeannie start-cluster +``` + +## Validation + +```bash +curl -I https://lab2025.duckdns.org/ +curl -I https://lab2025.duckdns.org/git/ +ssh ubuntu@132.145.170.74 ' + curl -I --max-time 5 http://192.168.100.240:80/ || true + curl -I --max-time 5 http://100.85.138.30:3000/ || true +' +``` + +Expected: + +- Public requests return `200`, `301`, `302`, or app-specific `404`. +- Edge backend curls do not fail with connection refused or timeout. diff --git a/docs/runbooks/gitea-failures.md b/docs/runbooks/gitea-failures.md new file mode 100644 index 0000000..a0ec4bc --- /dev/null +++ b/docs/runbooks/gitea-failures.md @@ -0,0 +1,123 @@ +# Gitea Failures Runbook + +Use this when Gitea is down, `/git/` returns `502`, Git push/pull fails, login +does not work, or the pipeline cannot access the repository. + +## Topology + +Gitea is intentionally outside Kubernetes: + +- Host: Debian `192.168.100.73` +- Tailscale IP: `100.85.138.30` +- Install directory: `/data/homelab-gitea` +- HTTP: `3000` +- SSH: `32222` +- Public path: `https://lab2025.duckdns.org/git/` + +## Quick Triage + +Run on the Debian host: + +```bash +cd ~/my-homelab-configs +./jeannie status +cd /data/homelab-gitea +sudo docker compose ps +sudo docker compose logs --tail=120 gitea +curl -I http://127.0.0.1:3000/ +tailscale ip -4 +``` + +Check the edge path: + +```bash +ssh ubuntu@132.145.170.74 'curl -I --max-time 5 http://100.85.138.30:3000/' +curl -I https://lab2025.duckdns.org/git/ +``` + +Check Git SSH from Debian: + +```bash +ssh -T -p 32222 git@192.168.100.73 || true +git remote -v +git ls-remote gitea main +``` + +## Common Cases + +Container is stopped: + +```bash +cd /data/homelab-gitea +sudo docker compose up -d +sudo docker compose ps +``` + +Docker or containerd is wedged: + +```bash +sudo systemctl restart containerd +sudo systemctl restart docker +cd /data/homelab-gitea +sudo docker compose up -d +``` + +Gitea works locally but not from OCI edge: + +```bash +tailscale ip -4 +ssh ubuntu@132.145.170.74 'tailscale status; curl -I --max-time 5 http://100.85.138.30:3000/' +``` + +If the Debian Tailscale IP changed, update the edge variable and reapply: + +```bash +tofu -chdir=bootstrap/edge apply +``` + +Git HTTPS prompts or fails authentication: + +- Prefer SSH for writes from the homelab. +- Keep the Debian working copy remote pointed at the LAN SSH endpoint: + +```bash +git remote set-url gitea ssh://git@192.168.100.73:32222/jv/my-homelab-configs.git +git branch --set-upstream-to=gitea/main main +``` + +Login password is lost: + +```bash +cd /data/homelab-gitea +sudo docker exec -it homelab-gitea gitea admin user list --config /data/gitea/conf/app.ini +sudo docker exec -it homelab-gitea gitea admin user change-password \ + --config /data/gitea/conf/app.ini \ + --username jv \ + --password '' +``` + +## Safe Redeploy + +Redeploying Gitea should not delete data because the Compose stack uses the +persistent data directory under `/data/homelab-gitea/data`. + +```bash +cd ~/my-homelab-configs +./jeannie deploy-gitea +./jeannie bootstrap-gitea-repo +``` + +## Validation + +```bash +curl -I http://127.0.0.1:3000/ +curl -I https://lab2025.duckdns.org/git/ +git ls-remote gitea main +sudo docker compose -f /data/homelab-gitea/docker-compose.yml ps +``` + +Expected: + +- Local HTTP returns a web response. +- Public `/git/` loads through edge. +- `git ls-remote` returns the `main` ref.