141 lines
3.1 KiB
Markdown
141 lines
3.1 KiB
Markdown
# Cluster Stop/Start Failures Runbook
|
|
|
|
Use this when `./jeannie stop-cluster` or `./jeannie start-cluster` does not
|
|
complete cleanly, RAM stays high after a stop, or the cluster does not recover
|
|
after a start.
|
|
|
|
## Intent
|
|
|
|
`stop-cluster` pauses Kubernetes without destroying kubeadm files, OpenTofu
|
|
state, PV data, or VM disks. It should keep host-level Docker Compose services
|
|
such as Gitea and RPi services separate from the Kubernetes runtime.
|
|
|
|
## Quick Triage
|
|
|
|
Run from the Debian host:
|
|
|
|
```bash
|
|
./jeannie status
|
|
systemctl is-active kubelet containerd docker
|
|
free -h
|
|
sudo crictl ps -a || true
|
|
kubectl get nodes -o wide || true
|
|
```
|
|
|
|
Check Pimox worker VM state:
|
|
|
|
```bash
|
|
ssh jv@192.168.100.80 'sudo qm list; sudo qm status 9010; sudo qm status 9011'
|
|
```
|
|
|
|
## Stop Failures
|
|
|
|
If Kubernetes containers remain after stop:
|
|
|
|
```bash
|
|
sudo systemctl stop kubelet
|
|
sudo crictl stop --all || true
|
|
sudo crictl rm --all || true
|
|
```
|
|
|
|
If RAM is still high, inspect what is actually using memory:
|
|
|
|
```bash
|
|
ps -eo pid,ppid,comm,%mem,rss,args --sort=-rss | head -40
|
|
sudo crictl ps -a || true
|
|
sudo docker ps
|
|
```
|
|
|
|
If the memory is from Kubernetes containers, remove stopped CRI containers:
|
|
|
|
```bash
|
|
sudo crictl rm --all || true
|
|
sudo crictl rmi --prune || true
|
|
```
|
|
|
|
If the memory is from Docker Compose services, do not stop Gitea or RPi DNS
|
|
unless that service is the incident. Those services intentionally stay online.
|
|
|
|
If worker VMs did not stop:
|
|
|
|
```bash
|
|
ssh jv@192.168.100.80 'sudo qm shutdown 9010 --timeout 120 || sudo qm stop 9010'
|
|
ssh jv@192.168.100.80 'sudo qm shutdown 9011 --timeout 120 || sudo qm stop 9011'
|
|
```
|
|
|
|
## Start Failures
|
|
|
|
Start the cluster runtime:
|
|
|
|
```bash
|
|
./jeannie start-cluster
|
|
```
|
|
|
|
If the API server does not come up:
|
|
|
|
```bash
|
|
sudo systemctl start containerd
|
|
sudo systemctl start kubelet
|
|
sudo journalctl -u kubelet -n 120 --no-pager
|
|
sudo crictl ps -a | head -50
|
|
```
|
|
|
|
If nodes stay `NotReady`:
|
|
|
|
```bash
|
|
kubectl get nodes -o wide
|
|
kubectl describe node debian | sed -n '/Conditions:/,/Addresses:/p'
|
|
kubectl -n kube-system get pods -o wide
|
|
```
|
|
|
|
Common causes:
|
|
|
|
- CNI pods are not running.
|
|
- containerd did not start on a worker.
|
|
- Pimox worker VMs are still stopped.
|
|
- Node clocks or DHCP addresses changed after restart.
|
|
|
|
Start Pimox workers manually if needed:
|
|
|
|
```bash
|
|
ssh jv@192.168.100.80 'sudo qm start 9010 || true; sudo qm start 9011 || true'
|
|
```
|
|
|
|
Then wait for nodes:
|
|
|
|
```bash
|
|
kubectl get nodes -w
|
|
```
|
|
|
|
## When To Rebuild Instead
|
|
|
|
Use rebuild only when start cannot recover a coherent cluster:
|
|
|
|
```bash
|
|
./jeannie rebuild-cluster
|
|
```
|
|
|
|
Prefer rebuild when:
|
|
|
|
- kubeadm state is inconsistent.
|
|
- kubelet versions drift from the control plane.
|
|
- worker VM disks/templates are known bad.
|
|
- repeated start attempts leave system pods permanently broken.
|
|
|
|
## Validation
|
|
|
|
```bash
|
|
kubectl get nodes -o wide
|
|
kubectl -n kube-system get pods -o wide
|
|
kubectl -n traefik get svc,pods -o wide
|
|
curl -I http://192.168.100.240/
|
|
./jeannie status
|
|
```
|
|
|
|
Expected:
|
|
|
|
- Control plane and desired workers are `Ready`.
|
|
- `kube-system` pods are running or completed.
|
|
- Traefik has the MetalLB IP.
|
|
- Host-level Gitea remains up throughout stop/start work.
|