2.8 KiB

Raw Blame History

Edge Failures Runbook

Use this when https://lab2025.duckdns.org/, /git/, or a subdomain returns 502 Bad Gateway, times out, or serves the wrong backend.

Topology

internet -> OCI jump box nginx -> HAProxy/Varnish/Squid -> homelab backend

Current important backends:

Website and Kubernetes apps: Traefik MetalLB IP 192.168.100.240:80
Gitea: Debian Tailscale IP 100.85.138.30:3000
OCI edge host: 132.145.170.74

Quick Triage

Run from the Debian homelab server:

./jeannie status
curl -I https://lab2025.duckdns.org/
curl -I https://lab2025.duckdns.org/git/
ssh ubuntu@132.145.170.74 'sudo docker compose -f /opt/homelab-edge/docker-compose.yml ps'

Check backend reachability from the OCI edge host:

ssh ubuntu@132.145.170.74 '
  curl -I --max-time 5 http://192.168.100.240:80/ || true
  curl -I --max-time 5 http://100.85.138.30:3000/ || true
  tailscale status
'

Interpret Results

/git/ works but / is 502:

Gitea path over Tailscale is healthy.
Traefik/MetalLB path is broken or the cluster is stopped.
Check Kubernetes services and MetalLB assignment.

/ works but /git/ is 502:

Edge to Traefik is healthy.
Gitea container, Debian Tailscale, or nginx /git/ proxy path is broken.
Use the Gitea runbook.

Both / and /git/ are 502:

Edge containers may be unhealthy, DuckDNS may point somewhere unexpected, or the OCI host may have lost Tailscale/private routing.

OCI can reach 100.85.138.30:3000 but public /git/ is broken:

Inspect /opt/homelab-edge/default.conf.
The /git/ location should proxy to Gitea without forwarding the /git/ prefix to the backend if Gitea is listening at /.

OCI cannot reach 192.168.100.240:80:

Check whether the cluster is intentionally stopped.
Check Traefik service and MetalLB IP assignment.
Check Tailscale subnet route acceptance on the OCI edge host.

Recovery

Refresh the edge stack from the repo:

tofu -chdir=bootstrap/edge apply

Restart edge containers only:

ssh ubuntu@132.145.170.74 '
  cd /opt/homelab-edge
  sudo docker compose restart
  sudo docker compose ps
'

Re-enable accepted Tailscale routes on the edge host:

ssh ubuntu@132.145.170.74 'sudo tailscale set --accept-routes=true'

If the cluster was intentionally stopped, either leave website paths down or start the cluster:

./jeannie start-cluster

Validation

curl -I https://lab2025.duckdns.org/
curl -I https://lab2025.duckdns.org/git/
ssh ubuntu@132.145.170.74 '
  curl -I --max-time 5 http://192.168.100.240:80/ || true
  curl -I --max-time 5 http://100.85.138.30:3000/ || true
'

Expected:

Public requests return 200, 301, 302, or app-specific 404.
Edge backend curls do not fail with connection refused or timeout.

2.8 KiB Raw Blame History