# Cluster Stop/Start Failures Runbook

Use this when `./jeannie stop-cluster` or `./jeannie start-cluster` does not
complete cleanly, RAM stays high after a stop, or the cluster does not recover
after a start.

## Intent

`stop-cluster` pauses Kubernetes without destroying kubeadm files, OpenTofu
state, PV data, or VM disks. It should keep host-level Docker Compose services
such as Gitea and RPi services separate from the Kubernetes runtime.

## Quick Triage

Run from the Debian host:

```bash
./jeannie status
systemctl is-active kubelet containerd docker
free -h
sudo crictl ps -a || true
kubectl get nodes -o wide || true
```

Check Pimox worker VM state:

```bash
ssh jv@192.168.100.80 'sudo qm list; sudo qm status 9010; sudo qm status 9011'
```

## Stop Failures

If Kubernetes containers remain after stop:

```bash
sudo systemctl stop kubelet
sudo crictl stop --all || true
sudo crictl rm --all || true
```

If RAM is still high, inspect what is actually using memory:

```bash
ps -eo pid,ppid,comm,%mem,rss,args --sort=-rss | head -40
sudo crictl ps -a || true
sudo docker ps
```

If the memory is from Kubernetes containers, remove stopped CRI containers:

```bash
sudo crictl rm --all || true
sudo crictl rmi --prune || true
```

If the memory is from Docker Compose services, do not stop Gitea or RPi DNS
unless that service is the incident. Those services intentionally stay online.

If worker VMs did not stop:

```bash
ssh jv@192.168.100.80 'sudo qm shutdown 9010 --timeout 120 || sudo qm stop 9010'
ssh jv@192.168.100.80 'sudo qm shutdown 9011 --timeout 120 || sudo qm stop 9011'
```

## Start Failures

Start the cluster runtime:

```bash
./jeannie start-cluster
```

If the API server does not come up:

```bash
sudo systemctl start containerd
sudo systemctl start kubelet
sudo journalctl -u kubelet -n 120 --no-pager
sudo crictl ps -a | head -50
```

If nodes stay `NotReady`:

```bash
kubectl get nodes -o wide
kubectl describe node debian | sed -n '/Conditions:/,/Addresses:/p'
kubectl -n kube-system get pods -o wide
```

Common causes:

- CNI pods are not running.
- containerd did not start on a worker.
- Pimox worker VMs are still stopped.
- Node clocks or DHCP addresses changed after restart.

Start Pimox workers manually if needed:

```bash
ssh jv@192.168.100.80 'sudo qm start 9010 || true; sudo qm start 9011 || true'
```

Then wait for nodes:

```bash
kubectl get nodes -w
```

## When To Rebuild Instead

Use rebuild only when start cannot recover a coherent cluster:

```bash
./jeannie rebuild-cluster
```

Prefer rebuild when:

- kubeadm state is inconsistent.
- kubelet versions drift from the control plane.
- worker VM disks/templates are known bad.
- repeated start attempts leave system pods permanently broken.

## Validation

```bash
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
kubectl -n traefik get svc,pods -o wide
curl -I http://192.168.100.240/
./jeannie status
```

Expected:

- Control plane and desired workers are `Ready`.
- `kube-system` pods are running or completed.
- Traefik has the MetalLB IP.
- Host-level Gitea remains up throughout stop/start work.