# Edge Failures Runbook

Use this when `https://lab2025.duckdns.org/`, `/git/`, or a subdomain returns
`502 Bad Gateway`, times out, or serves the wrong backend.

## Topology

```text
internet -> OCI jump box nginx -> HAProxy/Varnish/Squid -> homelab backend
```

Current important backends:

- Website and Kubernetes apps: Traefik MetalLB IP `192.168.100.240:80`
- Gitea: Debian Tailscale IP `100.85.138.30:3000`
- Heimdall: Debian Tailscale IP `100.85.138.30:8082`
- OCI edge host: `132.145.170.74`

## Quick Triage

Run from the Debian homelab server:

```bash
./jeannie status
curl -I https://lab2025.duckdns.org/
curl -I https://lab2025.duckdns.org/git/
ssh ubuntu@132.145.170.74 'sudo docker compose -f /opt/homelab-edge/docker-compose.yml ps'
```

Check backend reachability from the OCI edge host:

```bash
ssh ubuntu@132.145.170.74 '
  curl -I --max-time 5 http://192.168.100.240:80/ || true
  curl -I --max-time 5 http://100.85.138.30:3000/ || true
  curl -I --max-time 5 http://100.85.138.30:8082/ || true
  tailscale status
'
```

## Interpret Results

`/git/` works but `/` is `502`:

- Gitea path over Tailscale is healthy.
- Traefik/MetalLB path is broken or the cluster is stopped.
- Check Kubernetes services and MetalLB assignment.

`/` works but `/git/` is `502`:

- Edge to Traefik is healthy.
- Gitea container, Debian Tailscale, or nginx `/git/` proxy path is broken.
- Use the Gitea runbook.

Both `/` and `/git/` are `502`:

- Edge containers may be unhealthy, DuckDNS may point somewhere unexpected, or
  the OCI host may have lost Tailscale/private routing.

OCI can reach `100.85.138.30:3000` but public `/git/` is broken:

- Inspect `/opt/homelab-edge/default.conf`.
- The `/git/` location should proxy to Gitea without forwarding the `/git/`
  prefix to the backend if Gitea is listening at `/`.

OCI cannot reach `192.168.100.240:80`:

- Check whether the cluster is intentionally stopped.
- Check Traefik service and MetalLB IP assignment.
- Check Tailscale subnet route acceptance on the OCI edge host.

Only `heimdall.lab2025.duckdns.org` is broken:

- The dashboard is outside Kubernetes on the Debian host.
- Run `./jeannie deploy-heimdall`.
- Check `sudo docker compose -f /data/homelab-heimdall/docker-compose.yml ps`.

## Recovery

Refresh the edge stack from the repo:

```bash
./jeannie edge
```

Restart edge containers only:

```bash
ssh ubuntu@132.145.170.74 '
  cd /opt/homelab-edge
  sudo docker compose restart
  sudo docker compose ps
'
```

Re-enable accepted Tailscale routes on the edge host:

```bash
ssh ubuntu@132.145.170.74 'sudo tailscale set --accept-routes=true'
```

If the cluster was intentionally stopped, either leave website paths down or
start the cluster:

```bash
./jeannie start-cluster
```

## Validation

```bash
curl -I https://lab2025.duckdns.org/
curl -I https://lab2025.duckdns.org/git/
curl -I https://heimdall.lab2025.duckdns.org/
ssh ubuntu@132.145.170.74 '
  curl -I --max-time 5 http://192.168.100.240:80/ || true
  curl -I --max-time 5 http://100.85.138.30:3000/ || true
  curl -I --max-time 5 http://100.85.138.30:8082/ || true
'
```

Expected:

- Public requests return `200`, `301`, `302`, or app-specific `404`.
- Edge backend curls do not fail with connection refused or timeout.