Self-hosted Kubernetes delivery platform
Git push, validation, image build, registry, GitOps sync, policy guardrails, monitoring, retained storage, and VM worker provisioning in one small but operationally honest platform.
View architectureHomelab field notes
I accidentally built a tiny CI/CD platform
A case-study style walkthrough of how a Debian control plane, Pimox app workers, external Gitea, local registry, Kyverno policy, Argo CD, monitoring, and static demo shelf became a repeatable Kubernetes delivery path.
Portfolio case studies
Production-shaped evidence
These are the three proof points a hiring manager should see first: platform ownership, production reliability at scale, and the reserved MLOps path for model-serving work.
Enterprise SRE and incident automation
Oracle and prior enterprise roles show the production side: 20,000+ developer users, 10,000+ external customers, Linux troubleshooting, automation, runbooks, on-call improvement, and high-scale incident response.
View CV evidenceMLOps deployment platform placeholder
Reserved for the next serious demo: FastAPI inference, Kubernetes manifests, rollout strategy, model metrics, drift signals, and rollback behavior.
Open placeholderArchitecture map
The homelab, end to end
The current delivery path starts with a push to Gitea, runs local validation, builds arm64 images, syncs the validated commit into the GitOps mirror, and lets Argo CD reconcile from the app workers. The infrastructure path stays manual through lab.sh, including the PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, and the opt-in OpenWrt firewall VM, while the OCI edge routes public traffic back through the private path.
The diagram is intentionally operational: it shows the app delivery loop, image flow, provisioning path, worker-placement boundary, monitoring layer, OpenWrt firewall option, and public traffic path without hiding the practical bits that make a small lab behave like a platform.
Open the Christmas-tree version
Future me, judging
Be honest: why build all this instead of just running a couple containers like a normal person?
Me, holding coffee
Because apparently I looked at "host a website" and thought, "what if this had a control plane, GitOps, retained storage, an image registry, and several new ways to embarrass myself?" The real goal was practice: provision the infra, keep config in Git, deploy with automation, break it, fix it, and make sure I could rebuild it without relying on shell history and vibes.
Future me, judging
Why kubeadm? Were managed clusters too emotionally stable?
Me, holding coffee
Pretty much. kubeadm keeps the cluster close to the metal, which is a polite way of saying I get to see every sharp edge. The Debian node runs the control plane, the Raspberry Pi joins as an arm64 worker, and Pimox on an Orange Pi 5 Plus now gives me a repeatable way to add Debian 13 arm64 VM workers. Suddenly networking, storage, container runtimes, certs, and node recovery are not mysterious cloud magic. They are my problem.
Future me, judging
So where is the CI/CD part hiding?
Me, holding coffee
It is small, but it is real. OpenTofu brings up the cluster, platform, apps, and edge layers. Argo CD watches Git from the app workers, Kyverno keeps policy pressure on the workloads, Docker Buildx builds linux/arm64 images, and the local registry feeds the cluster. No enterprise dashboard fireworks, just a clean loop: Git changed, image built, cluster updated, nobody had to kubectl-edit anything at 2 AM.
Future me, judging
Why run your own registry and Gitea? Was the simple option unavailable?
Me, holding coffee
The simple option was very available, which is why I heroically ignored it. The registry means experiments do not need to go to a public image repo, and external Gitea gives the lab its own Git service without making Kubernetes responsible for its own source of truth. Together they make the setup feel less like "some containers under the stairs" and more like a tiny platform with opinions, responsibilities, and occasionally dramatic storage needs.
Future me, judging
What actually hurt the most?
Me, holding coffee
Storage. Always storage. Kubernetes, Docker, retained volumes, and build caches can fill a small root disk with the quiet confidence of a bad decision. Moving OpenEBS local volumes and Docker data to the external SSD turned the lab from "why is everything on fire?" into "okay, this is usable now." Growth, allegedly.
Future me, judging
So the platform controllers finally moved off the control plane?
Me, holding coffee
Yes. Argo CD and Kyverno now target the homelab.dev/node-role=app workers, including Kyverno hook jobs, so the Debian node can stay focused on control-plane duties. That one change made the lab feel less like everything was balanced on the first machine that happened to boot.
Future me, judging
Can the current cluster actually handle all that, or are we about to smoke the Pi?
Me, holding coffee
The Pi survives because the demos are intentionally local-first and now ship as a separate static artifact. The website pod stays a portfolio shell, the demos-static pod serves static bundles, and the user browser does the expensive work. If I later ship real ONNX object detection, Transformers.js, or full video transcoding models, those must lazy-load in the browser or move to a beefier node. The Raspberry Pi is brave, but it is not a GPU wearing a tiny hat.
Future me, judging
So the lab can now build its own worker nodes?
Me, holding coffee
Yes, and now with fewer crossed fingers. Debian runs a provisioning layer with dnsmasq, nginx, PXE boot files, GRUB, and a Debian 13 arm64 preseed. OpenTofu talks to Pimox through qm, creates VM 9000 on local storage, boots it from the network, installs the OS, runs golden-node prep, disables swap, verifies cgroups, installs containerd and kubeadm tooling, then seals it as a template. Worker clones are idempotent by VMID and now land on nvme_thin_pool, so local storage stays reserved for the template.
Future me, judging
And OpenWrt is joining the story too?
Me, holding coffee
Only as a simple firewall, not as a networking science project. The pipeline can create an opt-in OpenWrt ARM SystemReady VM, attach vmbr0 as WAN and vmbr1 as LAN, and configure the LAN side without rewriting Orange Pi host networking. DHCP stays optional, and VLANs wait until there is a managed switch and a local test window.
Future me, judging
What changed on the observability and scheduling side?
Me, holding coffee
Monitoring moved from "someday" to "running," and scheduling moved from "whatever fits" to explicit worker placement. Prometheus Stack, Grafana, Loki, Promtail, node-exporter, and kube-state-metrics give the lab useful signals; the next useful step is choosing the few alerts that would actually wake me up for the right reasons.
Recent activity log
What changed since the first build
The lab moved from a working Kubernetes experiment into a more complete self-hosted delivery system. The latest work focused on trust, repeatability, VM-based expansion, controller placement, and making deploys match the exact commit that passed validation.
- 01 Moved Gitea out of Kubernetes and onto the Raspberry Pi as the local Git service, while keeping the public /git/ route through the edge stack.
- 02 Installed and validated a Debian-hosted Gitea Actions runner so pushes to main can build, scan, and deploy without depending on a laptop session.
- 03 Added a custom checkout flow for the /git/ subpath and kept a persistent Debian checkout for the deployment scripts.
- 04 Added Gitleaks secret scanning and Trivy scanning for the app and infrastructure tree.
- 05 Changed deployment so the validated commit is pushed into the local GitOps mirror before lab.sh runs, preventing Argo CD from reconciling an older tree.
- 06 Hardened the website, demos-static, and registry workloads with non-root containers, read-only root filesystems, resource limits, and explicit writable volumes.
- 07 Split the demos into a dedicated demos-static image and Argo CD application so the PHP website stays small and boring.
- 08 Changed Gitea backups to dump from the Raspberry Pi Docker container and store archives on the Debian host.
- 09 Validated the full main-branch deployment path: fetch main, apply OpenTofu layers, build and push arm64 images, refresh Argo CD, and confirm the runner completes successfully.
- 10 Built the Debian 13 arm64 Pimox template end to end with PXE, preseed, qemu-guest-agent discovery, cgroup validation, swap disabled, and a final seal step.
- 11 Added NVMe-backed Pimox worker clone automation so VM 9000 stays on local storage while worker nodes are created on nvme_thin_pool.
- 12 Added an opt-in OpenWrt VM path for a simple firewall between vmbr0 and vmbr1, with guardrails that avoid Orange Pi host networking changes.
- 13 Installed the monitoring stack and moved platform add-ons such as Argo CD, Kyverno, and prometheus-stack work toward app-worker placement instead of treating the control plane as spare capacity.
Technologies and why they are here
- Debian Linux is the steady adult in the room: control plane host, deployment workstation, PXE/preseed server, and the place where OpenTofu, Docker, kubeadm, and the scripts do their thing.
- Raspberry Pi keeps the external Gitea service close to the lab, while Pimox on the Orange Pi 5 Plus provides the app-worker pool. The Debian template stays on local storage and Kubernetes worker clones go to nvme_thin_pool.
- OpenTofu makes the cluster, platform, apps, edge, and provisioning layers repeatable, because "I swear I remember the command" is not a disaster recovery strategy.
-
Calico handles pod networking, and OpenEBS hostpath storage keeps the important data around after rebuilds, because deleting everything by accident is only funny once.
Source:platform/main.tf
-
Argo CD is the GitOps referee and now runs on app workers: manifests live in Git, the cluster follows along, and manual drift gets side-eyed back into place.
Source:apps/main.tf
- The OCI edge host runs nginx, HAProxy, Varnish, and Squid so TLS, routing, and caching stay outside the home network while Tailscale sneaks the traffic back to the worker node.
- The shared theme toggle is plain CSS and JavaScript: dark mode uses an old-console green-on-black treatment, while light mode switches to black cursive on ivory across the website and demo catalog.
- The first demo keeps files in the browser. Image crunching uses native Canvas APIs today, while the fast serious path for video conversion is Rust compiled to WebAssembly with a TypeScript UI.
- The newer demos cover network jitter graphs, local JSON/JWT/log tools, an architecture simulator, an offline traveler converter, a redactor prototype, sentiment analysis, model-drift simulation, and the reserved MLOps platform page.
- The serious MLOps page is intentionally only a placeholder for now. The target design is a production-shaped inference demo with FastAPI, Kubernetes deployment, metrics, drift signals, canary or blue-green rollout, and rollback notes.
- The demo code now builds into its own demos-static image and Argo CD app, exposed at /demo-apps/. The PHP website only owns the catalog link, which is much less cursed.
- The Pimox worker pipeline uses qm over SSH to create an OVMF/virtio-scsi Debian 13 arm64 VM, wait for qemu-guest-agent, seal it, and convert VM 9000 into a reusable template on local storage.
- The golden image bakes in Kubernetes prerequisites: swap disabled, cgroup boot options checked, kernel modules loaded, containerd configured for systemd cgroups, kubeadm/kubelet/kubectl installed, and qemu-guest-agent enabled.
- Worker clone automation is count-based and idempotent: LAB_PIMOX_WORKER_COUNT=1 means ensure VM 9010 exists, not create a fresh worker on every run. New workers are refused if the target storage is local.
- OpenWrt is handled separately from the Debian golden-node template. The lab downloads the upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, and keeps it disabled unless LAB_OPENWRT_VM=true is set.
- The monitoring layer now includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, and kube-state-metrics, with placement guardrails so platform add-ons stop defaulting to the control plane.
Improvement backlog
Todo list for the next homelab pass
These are improvement proposals, not chores for the sake of chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, or makes the platform easier to operate when something fails.
- Move Gitea data from the Raspberry Pi SD card to SSD-backed storage.
- Keep the Debian bare GitOps mirror as the cluster source and add object-storage backups when OCI storage is ready.
- Add a real OpenTofu remote state backend with backup, locking, and a documented recovery path.
- Replace the remaining mutable latest image references with immutable tags or digest pins for demo workloads; the website image now uses a content-hash tag.
- Generate SBOMs and sign images so the local registry can prove what it is serving.
- Add Renovate or Dependabot-style dependency updates for base images, Helm charts, and GitHub/Gitea Actions.
- Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, and documented exceptions for platform components.
- Turn the installed observability stack into useful operations views: a few high-signal dashboards, alerts for node health, storage pressure, certificate expiry, and failed app syncs.
- Schedule backup restore drills for external Gitea and OpenEBS volumes, then write the exact restore runbook.
- Tighten TLS, SSH, and token rotation around the OCI edge, Gitea, registry, and runner credentials.
- Document the new storage split: local for the Pimox template, nvme_thin_pool for VM workers, OpenEBS for Kubernetes app data, and backup targets for anything that must survive a rebuild.
- Move sensitive app configuration into Sealed Secrets, External Secrets, or another explicit secret-management path.
- After the new disk has breathing room, clone the first Pimox worker from VM 9000, join it with kubeadm, and verify labels, taints, CNI, storage, and workload scheduling.
- Test the OpenWrt VM in a maintenance window before making it a gateway: confirm WAN, LAN, rollback access, DHCP settings, and that Pimox remains reachable.
- Buy or configure a managed switch before VLAN work. Until then, keep OpenWrt as a simple two-interface firewall and avoid risky remote bridge rewrites.
Visitor ideas
What would you improve next?
Send a practical idea for the homelab backlog. Submissions are stored as plain text, limited in size, and rendered escaped.