116 lines
2.8 KiB
Markdown
116 lines
2.8 KiB
Markdown
# Edge Failures Runbook
|
|
|
|
Use this when `https://lab2025.duckdns.org/`, `/git/`, or a subdomain returns
|
|
`502 Bad Gateway`, times out, or serves the wrong backend.
|
|
|
|
## Topology
|
|
|
|
```text
|
|
internet -> OCI jump box nginx -> HAProxy/Varnish/Squid -> homelab backend
|
|
```
|
|
|
|
Current important backends:
|
|
|
|
- Website and Kubernetes apps: Traefik MetalLB IP `192.168.100.240:80`
|
|
- Gitea: Debian Tailscale IP `100.85.138.30:3000`
|
|
- OCI edge host: `132.145.170.74`
|
|
|
|
## Quick Triage
|
|
|
|
Run from the Debian homelab server:
|
|
|
|
```bash
|
|
./jeannie status
|
|
curl -I https://lab2025.duckdns.org/
|
|
curl -I https://lab2025.duckdns.org/git/
|
|
ssh ubuntu@132.145.170.74 'sudo docker compose -f /opt/homelab-edge/docker-compose.yml ps'
|
|
```
|
|
|
|
Check backend reachability from the OCI edge host:
|
|
|
|
```bash
|
|
ssh ubuntu@132.145.170.74 '
|
|
curl -I --max-time 5 http://192.168.100.240:80/ || true
|
|
curl -I --max-time 5 http://100.85.138.30:3000/ || true
|
|
tailscale status
|
|
'
|
|
```
|
|
|
|
## Interpret Results
|
|
|
|
`/git/` works but `/` is `502`:
|
|
|
|
- Gitea path over Tailscale is healthy.
|
|
- Traefik/MetalLB path is broken or the cluster is stopped.
|
|
- Check Kubernetes services and MetalLB assignment.
|
|
|
|
`/` works but `/git/` is `502`:
|
|
|
|
- Edge to Traefik is healthy.
|
|
- Gitea container, Debian Tailscale, or nginx `/git/` proxy path is broken.
|
|
- Use the Gitea runbook.
|
|
|
|
Both `/` and `/git/` are `502`:
|
|
|
|
- Edge containers may be unhealthy, DuckDNS may point somewhere unexpected, or
|
|
the OCI host may have lost Tailscale/private routing.
|
|
|
|
OCI can reach `100.85.138.30:3000` but public `/git/` is broken:
|
|
|
|
- Inspect `/opt/homelab-edge/default.conf`.
|
|
- The `/git/` location should proxy to Gitea without forwarding the `/git/`
|
|
prefix to the backend if Gitea is listening at `/`.
|
|
|
|
OCI cannot reach `192.168.100.240:80`:
|
|
|
|
- Check whether the cluster is intentionally stopped.
|
|
- Check Traefik service and MetalLB IP assignment.
|
|
- Check Tailscale subnet route acceptance on the OCI edge host.
|
|
|
|
## Recovery
|
|
|
|
Refresh the edge stack from the repo:
|
|
|
|
```bash
|
|
tofu -chdir=bootstrap/edge apply
|
|
```
|
|
|
|
Restart edge containers only:
|
|
|
|
```bash
|
|
ssh ubuntu@132.145.170.74 '
|
|
cd /opt/homelab-edge
|
|
sudo docker compose restart
|
|
sudo docker compose ps
|
|
'
|
|
```
|
|
|
|
Re-enable accepted Tailscale routes on the edge host:
|
|
|
|
```bash
|
|
ssh ubuntu@132.145.170.74 'sudo tailscale set --accept-routes=true'
|
|
```
|
|
|
|
If the cluster was intentionally stopped, either leave website paths down or
|
|
start the cluster:
|
|
|
|
```bash
|
|
./jeannie start-cluster
|
|
```
|
|
|
|
## Validation
|
|
|
|
```bash
|
|
curl -I https://lab2025.duckdns.org/
|
|
curl -I https://lab2025.duckdns.org/git/
|
|
ssh ubuntu@132.145.170.74 '
|
|
curl -I --max-time 5 http://192.168.100.240:80/ || true
|
|
curl -I --max-time 5 http://100.85.138.30:3000/ || true
|
|
'
|
|
```
|
|
|
|
Expected:
|
|
|
|
- Public requests return `200`, `301`, `302`, or app-specific `404`.
|
|
- Edge backend curls do not fail with connection refused or timeout.
|