3.1 KiB

Raw Blame History

Cluster Stop/Start Failures Runbook

Use this when ./jeannie stop-cluster or ./jeannie start-cluster does not complete cleanly, RAM stays high after a stop, or the cluster does not recover after a start.

Intent

stop-cluster pauses Kubernetes without destroying kubeadm files, OpenTofu state, PV data, or VM disks. It should keep host-level Docker Compose services such as Gitea and RPi services separate from the Kubernetes runtime.

Quick Triage

Run from the Debian host:

./jeannie status
systemctl is-active kubelet containerd docker
free -h
sudo crictl ps -a || true
kubectl get nodes -o wide || true

Check Pimox worker VM state:

ssh jv@192.168.100.80 'sudo qm list; sudo qm status 9010; sudo qm status 9011'

Stop Failures

If Kubernetes containers remain after stop:

sudo systemctl stop kubelet
sudo crictl stop --all || true
sudo crictl rm --all || true

If RAM is still high, inspect what is actually using memory:

ps -eo pid,ppid,comm,%mem,rss,args --sort=-rss | head -40
sudo crictl ps -a || true
sudo docker ps

If the memory is from Kubernetes containers, remove stopped CRI containers:

sudo crictl rm --all || true
sudo crictl rmi --prune || true

If the memory is from Docker Compose services, do not stop Gitea or RPi DNS unless that service is the incident. Those services intentionally stay online.

If worker VMs did not stop:

ssh jv@192.168.100.80 'sudo qm shutdown 9010 --timeout 120 || sudo qm stop 9010'
ssh jv@192.168.100.80 'sudo qm shutdown 9011 --timeout 120 || sudo qm stop 9011'

Start Failures

Start the cluster runtime:

./jeannie start-cluster

If the API server does not come up:

sudo systemctl start containerd
sudo systemctl start kubelet
sudo journalctl -u kubelet -n 120 --no-pager
sudo crictl ps -a | head -50

If nodes stay NotReady:

kubectl get nodes -o wide
kubectl describe node debian | sed -n '/Conditions:/,/Addresses:/p'
kubectl -n kube-system get pods -o wide

Common causes:

CNI pods are not running.
containerd did not start on a worker.
Pimox worker VMs are still stopped.
Node clocks or DHCP addresses changed after restart.

Start Pimox workers manually if needed:

ssh jv@192.168.100.80 'sudo qm start 9010 || true; sudo qm start 9011 || true'

Then wait for nodes:

kubectl get nodes -w

When To Rebuild Instead

Use rebuild only when start cannot recover a coherent cluster:

./jeannie rebuild-cluster

Prefer rebuild when:

kubeadm state is inconsistent.
kubelet versions drift from the control plane.
worker VM disks/templates are known bad.
repeated start attempts leave system pods permanently broken.

Validation

kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
kubectl -n traefik get svc,pods -o wide
curl -I http://192.168.100.240/
./jeannie status

Expected:

Control plane and desired workers are Ready.
kube-system pods are running or completed.
Traefik has the MetalLB IP.
Host-level Gitea remains up throughout stop/start work.

3.1 KiB Raw Blame History