jv
/
my-homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
my-homelab-configs/README.md

571 lines
24 KiB
Markdown
Raw Blame History

# Homelab Kubernetes Pipeline
This cool repo bootstraps a hybrid kubeadm cluster and then hands app delivery to
Argo CD.
## Architecture
The lab is intentionally small but production-shaped:
- a Debian amd64 host runs the kubeadm control plane and local deployment tools
- a Raspberry Pi arm64 node runs selected workloads
- the Raspberry Pi also runs the always-on Gitea Docker service outside
Kubernetes
- the Debian host keeps a bare GitOps mirror under
`/home/jv/git-server/my-homelab-configs.git`
- a provisioning layer can PXE boot Debian 13 arm64 VMs for Pimox worker
templates
- OpenTofu owns the bootstrap layers for cluster, platform, apps, and edge
- Argo CD continuously reconciles Kubernetes manifests from this repo
- a local registry stores the website and demos images built for the worker
architecture
- SOPS with age is the committed secret-management path for future encrypted
Kubernetes secrets
- an OCI jump box provides the public edge path back into the homelab over
Tailscale
Run `./lab.sh up` and `./lab.sh nuke` only from the Debian homelab server. The
script intentionally refuses to run from non-Debian machines so a laptop cannot
accidentally modify the cluster.
## Flow
1. `bootstrap/provisioning`
- prepares a Debian server as a PXE and preseed service for arm64 VMs
- serves Debian 13 arm64 netboot assets through TFTP and HTTP
- creates a golden image install path with Kubernetes, containerd,
qemu-guest-agent, cloud-init, and storage client packages ready
- is driven by `./lab.sh up` when Pimox is reachable, without changing
Orange Pi host networking
2. `bootstrap/cluster`
- creates the kubeadm control plane on the Debian amd64 node
- joins worker nodes such as Raspberry Pi and Pimox Debian arm64 nodes
- configures Calico-compatible pod CIDR
- configures containerd to pull from the in-cluster NodePort registry
- creates retained host directories under `/var/openebs/local`
3. `bootstrap/platform`
- installs a minimal Calico deployment through the Tigera operator
- installs NodeLocal DNSCache for node-local DNS query caching
- can install MetalLB for LAN `LoadBalancer` services after an address pool
is chosen
- installs OpenEBS
- creates `openebs-hostpath-retain`
- installs Argo CD
- installs Kyverno with audit-first baseline Pod Security policies
- registers the private GitOps repo without storing the SSH private key in
Terraform state
4. `bootstrap/apps`
- registers Argo CD Applications from the `applications` map
- default apps are `container-registry`, `website-production`, and
`demos-static`
5. `bootstrap/edge`
- connects to the OCI jump box
- uploads nginx, HAProxy, Varnish, and Squid configs
- obtains and renews Let's Encrypt certificates for the configured hostname
- runs the edge cache/proxy chain with Docker Compose
## Prerequisites
On the Debian host:
- OpenTofu
- Docker with Buildx
- kubeadm, kubelet, kubectl, and containerd
- SSH access to worker nodes
- SSH access to the OCI edge host
- enough persistent storage for `/var/openebs/local` and `/var/lib/docker`
The default kubeconfig path is `/home/jv/.kube/config`. Override it with
`KUBECONFIG_PATH` or `TF_VAR_kubeconfig_path` when needed.
## Deploying
From the Debian server:
```bash
cd ~/my-homelab-configs
./lab.sh up
```
The script first deploys external Gitea to the Raspberry Pi with Docker Compose
so Git stays outside the Kubernetes rebuild blast radius. It then detects the
Pimox host at `192.168.100.80` in auto mode. When SSH, `qm`, and `vmbr0` are
available, it applies `bootstrap/provisioning`, creates or reuses the Debian 13
arm64 template, creates or reuses one worker VM clone, discovers the guest IP
through qemu-guest-agent, and passes that worker into the cluster layer. It then
applies the remaining OpenTofu stacks, refreshes Argo CD apps, waits for the
local registry, builds the website and demos images when their source changed,
pushes them to the registry, recreates pods only after a new image is built, and
applies the edge stack.
Set `LAB_PIMOX_PIPELINE=false` to skip Pimox automation. Set
`LAB_PIMOX_WORKER_COUNT=0` to create or refresh only the template. The pipeline
keeps the template on its configured `local` storage, creates new worker VM
clones on `nvme_thin_pool` by default, checks that the Pimox bridge already
exists, refuses `local` as worker clone storage, and refuses to edit Orange Pi
host networking.
`LAB_PIMOX_SKIP_WORKER_INDEXES` defaults to `1` because the first Pimox worker
slot was created manually. With the default `LAB_PIMOX_WORKER_COUNT=1`, the
pipeline keeps the template current and leaves VMID `9010` alone. Set
`LAB_PIMOX_SKIP_WORKER_INDEXES=''` if you want the pipeline to own the first
slot, or set `LAB_PIMOX_WORKER_COUNT=2` to manage the second slot while still
skipping the first.
OpenWrt firewall VM automation is available as a standalone command because it
attaches to both WAN and LAN bridges. Run `./lab.sh openwrt` after `vmbr1`
already exists on the Orange Pi. The pipeline downloads the OpenWrt ARM
SystemReady EFI image, writes basic WAN/LAN/firewall config into the image,
imports it as VM `9100`, attaches `vmbr0` as WAN and `vmbr1` as LAN, and stores
the VM disk on `nvme_thin_pool`. It leaves the VM stopped and not enabled for
host boot by default. It does not use the Debian Kubernetes golden-node template
for OpenWrt.
The website and demos images default to `linux/arm64` because both deployments
are pinned to the Raspberry Pi worker. Override with `WEBSITE_IMAGE_PLATFORMS`
or `DEMOS_IMAGE_PLATFORMS` only if node placement changes.
Build metadata is written under `.lab/` so repeat runs can skip the website
or demos image build when the source hash, platform, image reference, and
registry manifest still match.
Set `LAB_GITEA_DEPLOY=false` to skip the external Gitea deployment step when the
Raspberry Pi service is already managed manually. The default Gitea target is
`jv@192.168.100.89`, install directory `/opt/homelab-gitea`, HTTP port `3000`,
and SSH port `32222`.
## Validation
Useful checks after a rebuild:
```bash
export KUBECONFIG=/home/jv/.kube/config
kubectl get nodes
kubectl -n argocd get applications
kubectl -n container-registry get pods
kubectl -n website-production get pods -o wide
kubectl -n demos-static get pods -o wide
ssh jv@192.168.100.89 'cd /opt/homelab-gitea && sudo docker compose ps'
docker info --format '{{.DockerRootDir}}'
df -h / /var/openebs/local /var/lib/docker
```
The website should be reached through the configured public hostname, not the raw
OCI IP address, because the Let's Encrypt certificate is issued for the
hostname.
## Adding Nodes
For Pimox on Orange Pi 5 Plus, `./lab.sh up` can create the Debian 13 arm64
template and worker VM clones automatically. Defaults are intentionally tied to
the observed host: Pimox SSH host `192.168.100.80`, bridge `vmbr0`, template VMID
`9000` on `local` storage, two 4 GiB worker VMs starting at VMID `9010`, worker
clone storage `nvme_thin_pool`, and no CPU affinity because this Pimox is pinned
to Debian Bullseye. Details and override variables are in
`bootstrap/provisioning/README.md`.
Worker indexes are stable. Index `1` maps to VMID `9010`, node name
`pimox-worker-01`, and worker key `pimox01`; index `2` maps to VMID `9011`, and
so on. `LAB_PIMOX_SKIP_WORKER_INDEXES=1` leaves the first slot unmanaged while
allowing higher indexes to be automated.
Run a full cluster rebuild from the Debian server with:
```bash
./lab.sh rebuild-cluster
```
That path preserves external Raspberry Pi Gitea, rebuilds the Pimox template
with 2 cores and 4 GiB memory, replaces two Pimox worker VMs with 2 cores and
4 GiB memory, and joins those workers to the Kubernetes cluster. CPU affinity is
disabled by default because the Bullseye-pinned Pimox `qm` does not support it.
The Raspberry Pi worker is excluded by default while it hosts external Gitea.
To opt the Raspberry Pi back into the Kubernetes cluster, set
`LAB_INCLUDE_RASPBERRY_WORKER=true` or add entries to
`bootstrap/cluster/variables.tf` or a `.tfvars` file:
```hcl
worker_nodes = {
raspberrypi = {
host = "192.168.100.89"
user = "jv"
node_name = "raspberry"
ssh_key_path = "/home/jv/.ssh/id_ed25519"
}
}
```
Stateful apps currently pin retained local PVs to the `debian` node. Move or
duplicate those PV manifests when you want storage on another node.
## Workload Placement
`bootstrap/cluster` labels nodes with homelab placement metadata:
- `node-role.kubernetes.io/worker=worker` on every worker so `kubectl get nodes`
shows `worker` instead of `<none>` in the ROLES column
- `homelab.dev/node-role=control-plane` and `homelab.dev/storage=local` on the
Debian control plane
- `homelab.dev/node-role=edge-app` and `homelab.dev/storage=local` on the
Raspberry Pi worker
- `homelab.dev/node-role=app` and `homelab.dev/storage=nvme` on automated Pimox
worker clones
Override `control_plane_node_labels`, `worker_node_labels`,
`LAB_RASPBERRY_NODE_LABELS_JSON`, or `LAB_PIMOX_WORKER_NODE_LABELS_JSON` when
the physical layout changes. The current website, demos, and registry manifests
are not moved automatically because the public NodePort path and retained
OpenEBS hostpath PVs are node-local. Move workloads only after their storage and
edge path are ready on the target node. Gitea is outside Kubernetes and is moved
by changing the Raspberry Pi Docker install target instead.
The Prometheus stack control workloads are pinned to Pimox worker nodes by the
default `prometheus_stack_node_selector` (`homelab.dev/node-role=app` and
`homelab.dev/storage=nvme`). Because the Prometheus, Alertmanager, and Grafana
PVCs use retained local OpenEBS volumes, moving an existing install off the
Debian control plane requires discarding those PVCs. Run
`./lab.sh move-prometheus-stack-workers` from the Debian host to label existing
worker nodes, destroy only the existing `prometheus-stack` Helm release, delete
its retained PVC/PV objects, and recreate the stack on the worker selector.
The website and demos NodePorts are reachable from the OCI jump box through the
Raspberry Pi Tailscale interface. `bootstrap/cluster` installs a persistent
`homelab-tailscale-nodeport.service` on the configured worker to restore the
route, rp_filter settings, and iptables rules after reboot. Override the
defaults through `tailscale_nodeport_access` when the jump-box IP, Pi Tailscale
IP, pod CIDR, primary NodePort, or pod target port changes. Add any additional
public NodePorts to `tailscale_nodeport_extra_ports`:
```hcl
tailscale_nodeport_access = {
enabled = true
worker_key = "raspberrypi"
peer_ip = "100.118.255.19"
node_tailscale_ip = "100.77.80.72"
pod_cidr = "10.244.0.0/16"
node_port = 30080
target_port = 8080
}
tailscale_nodeport_extra_ports = [30081]
```
For `./lab.sh nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of
remote SSH targets when more worker nodes exist. Set it to an empty string for a
single-node rebuild.
## Adding Platform Tools
Add Helm releases through `bootstrap/platform`'s `extra_helm_releases` map.
## Policy Guardrails
`bootstrap/platform` installs Kyverno and the upstream baseline Pod Security
policies in `Audit` mode. This gives the lab policy reports for unsafe workload
settings without blocking existing pods during the first rollout. After reports
are clean, individual policies can be promoted to `Enforce` in
`bootstrap/platform/main.tf`.
## DNS Cache
`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with
`registry.k8s.io/dns/k8s-dns-node-cache`. The default listens on
`169.254.20.10` and the kube-dns service IP `10.96.0.10`, which keeps the
rollout compatible with the current kube-proxy iptables path without rewriting
kubelet DNS settings across the nodes. Override `nodelocal_dns` if the service
CIDR or upstream DNS servers change.
## MetalLB
MetalLB is present in `bootstrap/platform` but disabled by default. Enable it
only after reserving a LAN IP range outside DHCP and outside any future OpenWrt
LAN pool:
```bash
export TF_VAR_metallb='{
enabled = true
repository = "https://metallb.github.io/metallb"
version = "0.16.0"
namespace = "metallb-system"
address_pool = ["192.168.100.240-192.168.100.250"]
l2_advertisement_enabled = true
pool_name = "homelab-lan"
}'
```
The current website, demos, and registry services remain `NodePort` services
until the LAN address pool and edge route are tested manually. Gitea is not a
Kubernetes service; it runs on the Raspberry Pi Docker host.
## Secrets
Use SOPS with age for secrets that need to live in Git. Start from
`.sops.yaml.example`, replace the age recipient with the public key generated on
the Debian host, and commit the resulting `.sops.yaml`. Keep the private age key
outside the repo. Operational notes are in `docs/secrets.md`.
## Edge Services
The OCI jump box runs the public edge path:
```text
nginx -> HAProxy -> Varnish/Squid -> Raspberry Pi Tailscale NodePort
```
The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and
deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in
`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars`
file when the public host, SSH key, server name, backend Tailscale IP, or
NodePort changes.
The `/git/` route is intentionally different from the Kubernetes app routes: it
proxies to Gitea on the Raspberry Pi at the configured `backend_host` and
`gitea_backend_port` instead of a Kubernetes NodePort. This keeps public
read-only source browsing available even when the cluster has been destroyed.
Use the configured `server_name` in the browser, for example
`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser
certificate warning because the trusted certificate is issued for the hostname.
The edge stack uses HTTP-01 validation, so public DNS for `server_name` must
point to the OCI public IP and inbound TCP 80 and 443 must be open before
`./lab.sh up` runs. Set `TF_VAR_letsencrypt_email` to receive expiry notices,
or leave it empty to register without an email. Set
`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate.
## Adding Apps
Add Kubernetes manifests under `apps/<name>` and register them in
`bootstrap/apps`'s `applications` map. Argo CD will own sync, pruning, and
self-healing for the app.
## Storage
OpenEBS provides the platform storage provisioner. Stateful Kubernetes apps use
retained local PV paths such as `/var/openebs/local/registry`; these paths are
intentionally outside kubeadm reset paths so data can survive cluster
destroy/create cycles. Those critical volumes are declared explicitly as
retained local PVs so a rebuilt cluster binds back to the same host paths
instead of creating fresh directories.
For the current lab, `/var/openebs/local` and `/var/lib/docker` are expected to
live on larger storage than the root filesystem. This keeps retained PVs,
container layers, Buildx state, and image caches from filling `/`.
## Gitea
Gitea is external bootstrap infrastructure. It runs on the Raspberry Pi as an
always-on Docker Compose service from `infra/gitea/docker-compose.yml`, not as a
Kubernetes workload. This keeps Git available when the Kubernetes cluster is
destroyed and rebuilt.
The default data path is `/opt/homelab-gitea/data` on the Raspberry Pi SD card.
That is acceptable for the current temporary setup; move
`LAB_GITEA_INSTALL_DIR` to an SSD mount when the SSD is added.
Public source browsing stays available through
`https://lab2025.duckdns.org/git/`. Registration is disabled and anonymous users
can view public repositories, so the blog can link to code read-only while
writes still require an authenticated Gitea account.
The Debian bare repo remains the GitOps mirror:
```text
/home/jv/git-server/my-homelab-configs.git
```
Argo CD consumes that Debian mirror through the default `gitops_repo_url`.
Gitea Actions pushes the `main` commit into the mirror before running the
selected deploy command.
Deploy or refresh the external Gitea container from the Debian host with:
```bash
./lab.sh deploy-gitea
```
## Gitea Backups
`./lab.sh up` installs a Debian-host systemd timer named
`homelab-gitea-backup.timer`. The timer runs daily, SSHes to the Raspberry Pi,
executes `gitea dump` inside the Gitea Docker container, copies the dump back to
Debian, and stores it under `/home/jv/backups/gitea`. The default retention is
30 days.
The same install step also creates `homelab-gitea-restore-drill.timer`. The
monthly drill is non-destructive: it verifies the latest backup ZIP, extracts it
to a temporary directory, records a report under
`/home/jv/backups/gitea-restore-drills`, and removes the temporary extract. It
does not write into the live Raspberry Pi Gitea data directory.
Run a manual backup from the Debian server with:
```bash
./lab.sh backup-gitea
```
Run the restore drill manually with:
```bash
./lab.sh drill-gitea-restore
```
Useful checks:
```bash
systemctl list-timers homelab-gitea-backup.timer
systemctl list-timers homelab-gitea-restore-drill.timer
sudo systemctl start homelab-gitea-backup.service
ls -lh /home/jv/backups/gitea
ls -lh /home/jv/backups/gitea-restore-drills
```
## Gitea Actions
This repo includes a Gitea Actions workflow at
`.gitea/workflows/homelab-main.yml`. It runs only on pushes to `main` and targets
a repository-scoped Debian host runner with the label `homelab-debian`.
The workflow only blocks automatic deploy for Raspberry Pi Gitea service
changes: files under `infra/gitea/`, or edits inside the `deploy_gitea`,
`install_gitea_backup_timer`, `backup_gitea`, or `drill_gitea_restore`
functions in `lab.sh`. Other changes use `HOMELAB_ACTION_COMMAND=auto` by
default: Actions runs `./lab.sh apps` when the Debian runner already has a
cluster kubeconfig, or `./lab.sh rebuild-cluster` when it does not.
Set `HOMELAB_ACTION_COMMAND=apps` or `HOMELAB_ACTION_COMMAND=rebuild-cluster`
on the runner to force one path.
`./lab.sh bootstrap-gitea-repo` also registers the Debian host SSH public key
with the Gitea repository and switches the Debian working copy's `gitea` remote
to `ssh://git@192.168.100.89:32222/jv/my-homelab-configs.git`. The default key
is `/home/jv/.ssh/id_ed25519.pub`; set `LAB_GITEA_REPO_SSH_KEY_PATH` to use a
different Debian-host key, or `LAB_GITEA_REPO_SSH_BOOTSTRAP=false` to leave SSH
access unchanged. The Actions deploy job uses the checked-out Actions workspace
as the source commit, updates the first available persistent checkout from
`HOMELAB_DEPLOY_DIR`, `/home/jv/my-homelab-configs`, or
`/home/jv/repos/my-homelab-configs`, and otherwise deploys directly from the
Actions workspace. It does not need SSH read access back to Gitea.
Enable Actions for the repository in Gitea, then create a repository-level runner
token from:
```text
https://lab2025.duckdns.org/git/jv/my-homelab-configs/settings/actions/runners
```
Register and start the Debian runner from the Debian server:
```bash
cd ~/my-homelab-configs
GITEA_RUNNER_REGISTRATION_TOKEN='<repo-runner-token>' ./lab.sh install-gitea-runner
```
The runner is installed as `homelab-gitea-runner.service`, runs as user `jv`, and
uses a host label instead of a Docker job container because deployment needs the
Debian host's Docker, OpenTofu, kubeconfig, SSH keys, and local state.
The deployment job is non-interactive. User `jv` must be able to run `sudo -n
true` on the Debian host for deployment commands that require sudo.
Useful checks:
```bash
systemctl status homelab-gitea-runner.service
journalctl -u homelab-gitea-runner.service -n 100 --no-pager
```
## Renovate
`renovate.json` defines dependency update rules for Dockerfiles, OpenTofu
providers, Helm chart versions, and the pinned tools used by the Gitea Actions
workflow. Renovate should open reviewable update branches or PRs only; it must
not auto-merge infrastructure changes. Keep app-only dependency updates on the
normal Gitea Actions path, and run `./lab.sh up` manually on the Debian server
for platform or provisioning updates.
## Destructive Rebuilds
`./lab.sh nuke` resets kubeadm, containerd runtime state, CNI files, Calico
links, iptables rules, and local OpenTofu state. It does not delete retained data
under `/var/openebs/local`.
For multi-node labs, set `WORKER_SSH_TARGETS` to a space-separated list of SSH
targets. It defaults to an empty string so the Raspberry Pi Gitea host is not
cleaned unless you explicitly include it.
## Website App
The website is a PHP app under `apps/website`. It includes a home page, CV page,
blog page, and demos page, plus a lightweight translation flow backed by Ollama.
Static language files live in `apps/website/lang`; unsupported browser languages
can be translated by the client and saved through `save_lang.php` as runtime
JSON data on the website PVC.
The CV page has two client-side presentation modes:
- `Elegant`: dark, minimal, terminal-inspired styling with a square profile
image and light green console text.
- `Fancy`: centered circular profile image, cursive orbit text, and a
cursor-following portrait rotation effect.
The Demos page is a catalog in the PHP website. The actual demo applications are
served from a separate `demos-static` artifact under `apps/demos-static` and are
published through the `demos-static` Argo CD application. Public traffic reaches
them through the edge path at `/demo-apps/`.
`./lab.sh up` builds and pushes two independent images:
- `php-website:latest` from `apps/website`
- `demos-static:latest` from `apps/demos-static`
The first demo, `The Client-Side Media Cruncher (Wasm + TS)`, currently performs
private, browser-only image compression and conversion using native Canvas APIs.
Heavier video conversion, such as MP4 to WebM, should use a Rust core compiled
to WebAssembly with a TypeScript UI so the codec work stays fast and still
avoids backend uploads.
The demos are designed to be local-first so the current cluster can serve them
from the Raspberry Pi worker without turning either pod into an application
server. The website pod serves the portfolio shell and the `demos-static` pod
serves static demo bundles; CPU-heavy work runs in the visitor's browser. With
the current deployments pinned to the Raspberry Pi, avoid bundling large ML
models, server-side WebSocket probes, or backend video transcoders into either
image. If those demos become production-grade, lazy load model assets in the
browser or move backend workers to a larger node, such as VMs on the Orange Pi 5
Plus.
Current demo inventory:
- Client-side media cruncher: image conversion/compression with Canvas; future
Rust/Wasm codec path for video.
- Internet quality visualizer: live Canvas graph for latency, jitter, and
stability using same-origin browser probes; a dedicated WebSocket echo endpoint
would be the production version.
- Local log and JSON toolbelt: JSON formatting, JWT decoding, URL parsing, and
local text-log filtering.
- Architecture simulator: click-driven load, crash, and auto-scale simulation.
- Offline traveler converter: PWA shell with timezone, currency, and GB/GiB
conversions.
- Privacy-first redactor: local image redaction prototype; future
onnxruntime-web plus quantized YOLO or face model path.
- Local sentiment sandbox: lightweight local sentiment, keyword, and summary
prototype; future Transformers.js/ONNX path.
- Model drift simulator: visual MLOps playground for spikes, corrupted inputs,
and retraining.
The Kubernetes deployment uses `apps/website/web-app.yaml`. Keep the image
reference there aligned with `TF_VAR_registry_endpoint`, because `lab.sh` derives
the registry endpoint from that manifest.
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
make bootstrap behavior reproducible across nodes and rebuilds.
Reference in New Issue View Git Blame Copy Permalink