jv
/
my-homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Adding traefik to the mix
Homelab Main / deploy (push) Successful in 1m55s Details

This commit is contained in:
juvdiaz 2026-06-02 13:59:15 -06:00
parent ccb8247577
commit ad81d37119
16 changed files with 397 additions and 133 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96
README.md
View File

@ -48,8 +48,8 @@ accidentally modify the cluster.
3. `bootstrap/platform`
- installs a minimal Calico deployment through the Tigera operator
- installs NodeLocal DNSCache for node-local DNS query caching
- can install MetalLB for LAN `LoadBalancer` services after an address pool
is chosen
- installs MetalLB for LAN `LoadBalancer` services
- installs Traefik as the single Kubernetes ingress entry point
- installs OpenEBS
- creates `openebs-hostpath-retain`
- installs Argo CD
@ -245,29 +245,58 @@ you intentionally accept losing that monitoring data. A planned monitoring data
migration should be handled as a separate maintenance task with backup,
delete/recreate or storage migration steps, and post-restore checks.
The website and demos NodePorts are reachable from the OCI jump box through the
Raspberry Pi Tailscale interface. `bootstrap/cluster` installs a persistent
`homelab-tailscale-nodeport.service` on the configured worker to restore the
route, rp_filter settings, and iptables rules after reboot. Override the
defaults through `tailscale_nodeport_access` when the jump-box IP, Pi Tailscale
IP, pod CIDR, primary NodePort, or pod target port changes. Add any additional
public NodePorts to `tailscale_nodeport_extra_ports`:
The older NodePort path is now reserved for special cases such as the local
registry. `bootstrap/cluster` still contains `homelab-tailscale-nodeport`
support, but app traffic should normally enter through Traefik's MetalLB
address instead of per-app NodePorts. The cluster stack advertises the LAN
subnet from the configured Tailscale worker so the OCI edge can route to the
Traefik LoadBalancer address:
```hcl
tailscale_nodeport_access = {
metallb = {
enabled = true
worker_key = "raspberrypi"
peer_ip = "100.118.255.19"
node_tailscale_ip = "100.77.80.72"
pod_cidr = "10.244.0.0/16"
node_port = 30080
target_port = 8080
repository = "https://metallb.github.io/metallb"
version = "0.16.0"
namespace = "metallb-system"
address_pool = ["192.168.100.240-192.168.100.240"]
l2_advertisement_enabled = true
pool_name = "homelab-lan"
}
tailscale_nodeport_extra_ports = [30081, 30082, 30083, 30084, 30085, 30086]
tailscale_nodeport_extra_target_ports = [80, 3000, 9090, 9093]
traefik = {
enabled = true
repository = "https://helm.traefik.io/traefik"
chart_version = "40.2.0"
namespace = "traefik"
load_balancer_ip = "192.168.100.240"
ingress_class = "traefik"
}
tailscale_subnet_routes = {
enabled = true
worker_key = "raspberrypi"
routes = ["192.168.100.0/24"]
}
```
DuckDNS resolves `*.lab2025.duckdns.org` to the OCI edge, so public requests for
the service hostnames land on the same edge host. For direct LAN testing, point
LAN DNS, `/etc/hosts`, or a Tailscale DNS override for app hostnames at
Traefik's address:
```text
192.168.100.240 lab2025.duckdns.org
192.168.100.240 demos.lab2025.duckdns.org
192.168.100.240 heimdall.lab2025.duckdns.org
192.168.100.240 grafana.lab2025.duckdns.org
192.168.100.240 argocd.lab2025.duckdns.org
```
The edge stack uses Traefik as its backend by default and validates
`http://192.168.100.240:80/` before updating the edge containers. If Tailscale
subnet-route approval is not automatic in the tailnet policy, the edge deploy
will fail clearly instead of silently keeping a broken public path.
For `./lab.sh nuke`, set `WORKER_SSH_TARGETS` to a space-separated list of
remote SSH targets when more worker nodes exist. Set it to an empty string for a
single-node rebuild.
@ -311,9 +340,11 @@ export TF_VAR_metallb='{
}'
```
The current website, demos, and registry services remain `NodePort` services
until the LAN address pool and edge route are tested manually. Gitea is not a
Kubernetes service; it runs on the Raspberry Pi Docker host.
Traefik uses MetalLB for a LAN `LoadBalancer` address. App services such as the
website, demos, and Heimdall should be `ClusterIP` services behind Kubernetes
Ingress objects. The local registry remains a `NodePort` because the cluster
nodes use it as a pull endpoint. Gitea is not a Kubernetes service; it runs on
the Raspberry Pi Docker host.
## Secrets
@ -327,28 +358,29 @@ outside the repo. Operational notes are in `docs/secrets.md`.
The OCI jump box runs the public edge path:
```text
nginx -> HAProxy -> Varnish/Squid -> Raspberry Pi Tailscale NodePort
nginx -> HAProxy -> Varnish/Squid -> Traefik MetalLB IP
```
The `bootstrap/edge` stack renders configs from `bootstrap/edge/templates` and
deploys them to `/opt/homelab-edge` on the OCI host. Defaults are in
`bootstrap/edge/variables.tf`; override them through `TF_VAR_*` or a `.tfvars`
file when the public host, SSH key, server name, backend Tailscale IP, or
NodePort changes.
Traefik backend address changes.
The `/git/` route is intentionally different from the Kubernetes app routes: it
proxies to Gitea on the Raspberry Pi at the configured `backend_host` and
`gitea_backend_port` instead of a Kubernetes NodePort. This keeps public
read-only source browsing available even when the cluster has been destroyed.
proxies to Gitea on the Raspberry Pi at the configured `gitea_backend_port`
instead of Traefik. This keeps public read-only source browsing available even
when the cluster has been destroyed.
Use the configured `server_name` in the browser, for example
`https://lab2025.duckdns.org`. A raw OCI IP address will still show a browser
certificate warning because the trusted certificate is issued for the hostname.
The edge stack uses HTTP-01 validation, so public DNS for `server_name` must
point to the OCI public IP and inbound TCP 80 and 443 must be open before
`./lab.sh up` runs. Set `TF_VAR_letsencrypt_email` to receive expiry notices,
or leave it empty to register without an email. Set
The edge stack uses HTTP-01 validation and requests one certificate covering
`server_name` plus `additional_server_names`. DuckDNS resolves sub-subdomains
under `lab2025.duckdns.org` to the same edge IP, so inbound TCP 80 and 443 must
be open before `./lab.sh up` runs. Set `TF_VAR_letsencrypt_email` to receive
expiry notices, or leave it empty to register without an email. Set
`TF_VAR_enable_letsencrypt=false` to keep using the temporary local certificate.
## Adding Apps
@ -362,8 +394,8 @@ It runs the LinuxServer.io Heimdall dashboard, persists `/config` on
OpenEBS, and seeds tiles for the website, demo apps, Gitea, Grafana,
Prometheus, Alertmanager, Argo CD, the local registry, and Heimdall itself.
Because Heimdall does not support reverse-proxy subfolder hosting cleanly, it
is exposed through NodePort `30082`; the dashboard's internal tool links use
the Raspberry Pi Tailscale NodePort address.
is exposed through the dedicated hostname `heimdall.lab2025.duckdns.org` rather
than a `/heimdall/` path.
## Storage

5
apps/demos-static/kustomization.yaml Normal file
View File

@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- web-app.yaml

22
apps/demos-static/web-app.yaml
View File

@ -75,11 +75,27 @@ metadata:
name: demos-static-service
namespace: demos-static
spec:
type: NodePort
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: http
nodePort: 30081
selector:
app: demos-static
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demos-static
namespace: demos-static
spec:
ingressClassName: traefik
rules:
- host: demos.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: demos-static-service
port:
number: 80

96
apps/heimdall/ingress.yaml Normal file
View File

@ -0,0 +1,96 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: heimdall
namespace: heimdall
spec:
ingressClassName: traefik
rules:
- host: heimdall.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: heimdall
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: monitoring
spec:
ingressClassName: traefik
rules:
- host: grafana.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-stack-grafana
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus
namespace: monitoring
spec:
ingressClassName: traefik
rules:
- host: prometheus.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-stack-kube-prom-prometheus
port:
number: 9090
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: alertmanager
namespace: monitoring
spec:
ingressClassName: traefik
rules:
- host: alertmanager.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-stack-kube-prom-alertmanager
port:
number: 9093
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-server
namespace: argocd
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: https
spec:
ingressClassName: traefik
rules:
- host: argocd.lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 443

2
apps/heimdall/kustomization.yaml
View File

@ -3,4 +3,4 @@ kind: Kustomization
resources:
- namespace.yaml
- web-app.yaml
- ui-nodeports.yaml
- ingress.yaml

62
apps/heimdall/ui-nodeports.yaml
View File

@ -1,62 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: grafana-nodeport
namespace: monitoring
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 3000
nodePort: 30083
selector:
app.kubernetes.io/instance: prometheus-stack
app.kubernetes.io/name: grafana
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-nodeport
namespace: monitoring
spec:
type: NodePort
ports:
- name: http
port: 9090
targetPort: 9090
nodePort: 30084
selector:
app.kubernetes.io/name: prometheus
operator.prometheus.io/name: prometheus-stack-kube-prom-prometheus
---
apiVersion: v1
kind: Service
metadata:
name: alertmanager-nodeport
namespace: monitoring
spec:
type: NodePort
ports:
- name: http
port: 9093
targetPort: 9093
nodePort: 30085
selector:
app.kubernetes.io/name: alertmanager
alertmanager: prometheus-stack-kube-prom-alertmanager
---
apiVersion: v1
kind: Service
metadata:
name: argocd-server-nodeport
namespace: argocd
spec:
type: NodePort
ports:
- name: https
port: 443
targetPort: 8080
nodePort: 30086
selector:
app.kubernetes.io/name: argocd-server

14
apps/heimdall/web-app.yaml
View File

@ -28,7 +28,7 @@ data:
},
{
"title": "Demo Apps",
"url": "https://lab2025.duckdns.org/demo-apps/",
"url": "http://demos.lab2025.duckdns.org/",
"description": "Static browser-side demo catalog",
"colour": "#2563eb",
"class": "Demos"
@ -42,28 +42,28 @@ data:
},
{
"title": "Grafana",
"url": "http://100.77.80.72:30083/",
"url": "http://grafana.lab2025.duckdns.org/",
"description": "Monitoring dashboards",
"colour": "#f97316",
"class": "Grafana"
},
{
"title": "Prometheus",
"url": "http://100.77.80.72:30084/",
"url": "http://prometheus.lab2025.duckdns.org/",
"description": "Prometheus query UI",
"colour": "#dc2626",
"class": "Prometheus"
},
{
"title": "Alertmanager",
"url": "http://100.77.80.72:30085/",
"url": "http://alertmanager.lab2025.duckdns.org/",
"description": "Alert routing and silences",
"colour": "#b91c1c",
"class": "Alertmanager"
},
{
"title": "Argo CD",
"url": "https://100.77.80.72:30086/",
"url": "http://argocd.lab2025.duckdns.org/",
"description": "GitOps application sync status",
"colour": "#0ea5e9",
"class": "ArgoCD"
@ -77,7 +77,7 @@ data:
},
{
"title": "Heimdall",
"url": "http://100.77.80.72:30082/",
"url": "http://heimdall.lab2025.duckdns.org/",
"description": "Homelab service dashboard",
"colour": "#7c3aed",
"class": "Heimdall"
@ -282,10 +282,8 @@ metadata:
name: heimdall
namespace: heimdall
spec:
type: NodePort
ports:
- port: 80
targetPort: http
nodePort: 30082
selector:
app: heimdall

22
apps/website/web-app.yaml
View File

@ -119,11 +119,27 @@ metadata:
name: php-website-service
namespace: website-production
spec:
type: NodePort
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: http
nodePort: 30080
selector:
app: php-website
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: php-website
namespace: website-production
spec:
ingressClassName: traefik
rules:
- host: lab2025.duckdns.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: php-website-service
port:
number: 80

28
bootstrap/cluster/main.tf
View File

@ -352,6 +352,9 @@ resource "null_resource" "kubeadm_worker" {
tailscale_nodeport_pod_cidr = var.tailscale_nodeport_access.pod_cidr
tailscale_nodeport_node_ports = join(" ", distinct(concat([var.tailscale_nodeport_access.node_port], var.tailscale_nodeport_extra_ports)))
tailscale_nodeport_target_ports = join(" ", distinct(concat([var.tailscale_nodeport_access.target_port], var.tailscale_nodeport_extra_target_ports)))
tailscale_subnet_routes_version = "1"
tailscale_subnet_routes_enabled = var.tailscale_subnet_routes.enabled && each.key == var.tailscale_subnet_routes.worker_key ? "true" : "false"
tailscale_subnet_routes = join(",", var.tailscale_subnet_routes.routes)
}
connection {
@ -695,6 +698,31 @@ configure_tailscale_nodeport_access \
"${self.triggers.tailscale_nodeport_node_ports}" \
"${self.triggers.tailscale_nodeport_target_ports}"
configure_tailscale_subnet_routes() {
local enabled="$1"
local routes="$2"
if [ "$enabled" != "true" ]; then
return 0
fi
if ! command -v tailscale >/dev/null 2>&1; then
echo "tailscale is required to advertise subnet routes but is not installed on this worker." >&2
exit 1
fi
if [ -z "$routes" ]; then
echo "tailscale subnet route advertisement is enabled but no routes were configured." >&2
exit 1
fi
sudo tailscale set --advertise-routes="$routes"
}
configure_tailscale_subnet_routes \
"${self.triggers.tailscale_subnet_routes_enabled}" \
"${self.triggers.tailscale_subnet_routes}"
configure_containerd_registry "${self.triggers.registry_endpoint}"
pv_dirs="${self.triggers.persistent_volume_dirs}"

20
bootstrap/cluster/variables.tf
View File

@ -81,7 +81,7 @@ variable "tailscale_nodeport_access" {
})
default = {
enabled = true
enabled = false
worker_key = "raspberrypi"
peer_ip = "100.118.255.19"
node_tailscale_ip = "100.77.80.72"
@ -93,10 +93,24 @@ variable "tailscale_nodeport_access" {
variable "tailscale_nodeport_extra_ports" {
type = list(number)
default = [30081, 30082, 30083, 30084, 30085, 30086]
default = []
}
variable "tailscale_nodeport_extra_target_ports" {
type = list(number)
default = [80, 3000, 9090, 9093]
default = []
}
variable "tailscale_subnet_routes" {
type = object({
enabled = bool
worker_key = string
routes = list(string)
})
default = {
enabled = true
worker_key = "raspberrypi"
routes = ["192.168.100.0/24"]
}
}

18
bootstrap/edge/main.tf
View File

@ -10,8 +10,10 @@ terraform {
locals {
compose_file = templatefile("${path.module}/templates/docker-compose.yml.tftpl", {})
server_names = distinct(concat([var.server_name], var.additional_server_names))
default_conf = templatefile("${path.module}/templates/default.conf.tftpl", {
server_name = var.server_name
server_names = join(" ", local.server_names)
backend_host = var.backend_host
demos_backend_port = var.demos_backend_port
gitea_backend_port = var.gitea_backend_port
@ -43,6 +45,10 @@ resource "null_resource" "edge_services" {
user = var.edge_user
install_dir = var.edge_install_dir
server_name = var.server_name
server_names = join(" ", local.server_names)
certbot_domain_args = join(" ", [for name in local.server_names : "-d ${name}"])
backend_host = var.backend_host
backend_port = tostring(var.backend_port)
enable_letsencrypt = tostring(var.enable_letsencrypt)
letsencrypt_email = var.letsencrypt_email
letsencrypt_staging = tostring(var.letsencrypt_staging)
@ -96,6 +102,10 @@ set -eu
install_dir="${self.triggers.install_dir}"
server_name="${self.triggers.server_name}"
server_names="${self.triggers.server_names}"
certbot_domain_args="${self.triggers.certbot_domain_args}"
backend_host="${self.triggers.backend_host}"
backend_port="${self.triggers.backend_port}"
enable_letsencrypt="${self.triggers.enable_letsencrypt}"
letsencrypt_email="${self.triggers.letsencrypt_email}"
letsencrypt_staging="${self.triggers.letsencrypt_staging}"
@ -116,6 +126,12 @@ install_missing_packages() {
install_missing_packages ca-certificates curl openssl
if ! curl -fsS --connect-timeout 10 -H "Host: $server_name" "http://$backend_host:$backend_port/" >/dev/null; then
echo "Cannot reach Traefik backend http://$backend_host:$backend_port/ for $server_name from the edge host." >&2
echo "Check that MetalLB assigned the Traefik LoadBalancer IP and that the edge host has a route to it." >&2
exit 1
fi
if ! command -v docker >/dev/null 2>&1; then
curl -fsSL https://get.docker.com | sudo sh
fi
@ -243,7 +259,7 @@ if [ "$enable_letsencrypt" = "true" ]; then
"$certbot_image" certonly \
--webroot \
-w /var/www/certbot \
-d "$server_name" \
$certbot_domain_args \
--preferred-challenges http \
--agree-tos \
--non-interactive \

18
bootstrap/edge/templates/default.conf.tftpl
View File

@ -31,7 +31,7 @@ real_ip_header CF-Connecting-IP;
server {
listen 80;
server_name ${server_name};
server_name ${server_names};
location ^~ /.well-known/acme-challenge/ {
root /var/www/certbot;
@ -46,7 +46,7 @@ server {
server {
listen 443 ssl;
server_name ${server_name};
server_name ${server_names};
ssl_certificate /etc/nginx/certs/current.crt;
ssl_certificate_key /etc/nginx/certs/current.key;
@ -97,19 +97,7 @@ server {
}
location ^~ /demo-apps/ {
limit_req zone=one burst=20 nodelay;
proxy_pass http://${backend_host}:${demos_backend_port}/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header CF-Connecting-IP $http_cf_connecting_ip;
proxy_cache static_assets;
proxy_cache_valid 200 301 302 15m;
proxy_cache_key "$scheme$request_method$host$request_uri";
add_header X-Nginx-Cache "$upstream_cache_status";
return 301 https://demos.${server_name}/;
}
location ~* \.(css|js)$ {

16
bootstrap/edge/variables.tf
View File

@ -23,6 +23,18 @@ variable "server_name" {
default = "lab2025.duckdns.org"
}
variable "additional_server_names" {
type = list(string)
default = [
"demos.lab2025.duckdns.org",
"heimdall.lab2025.duckdns.org",
"grafana.lab2025.duckdns.org",
"prometheus.lab2025.duckdns.org",
"alertmanager.lab2025.duckdns.org",
"argocd.lab2025.duckdns.org",
]
}
variable "enable_letsencrypt" {
type = bool
default = true
@ -40,12 +52,12 @@ variable "letsencrypt_staging" {
variable "backend_host" {
type = string
default = "100.77.80.72"
default = "192.168.100.240"
}
variable "backend_port" {
type = number
default = 30080
default = 80
}
variable "demos_backend_port" {

83
bootstrap/platform/main.tf
View File

@ -115,6 +115,10 @@ EOT
"kubernetes.io/os" = "linux"
"homelab.dev/workload-class" = "platform"
}
traefik_node_selector = {
"kubernetes.io/os" = "linux"
"homelab.dev/workload-class" = "platform"
}
argocd_component_label_values = {
application_set = "argocd-applicationset-controller"
@ -753,6 +757,85 @@ EOT
}
}
resource "helm_release" "traefik" {
for_each = var.traefik.enabled ? { enabled = true } : {}
depends_on = [null_resource.metallb_l2_config]
name = "traefik"
repository = var.traefik.repository
chart = "traefik"
version = var.traefik.chart_version
namespace = var.traefik.namespace
create_namespace = true
timeout = 600
wait = true
values = [
yamlencode({
deployment = {
replicas = 1
}
nodeSelector = local.traefik_node_selector
ingressClass = {
enabled = true
isDefaultClass = true
name = var.traefik.ingress_class
}
gateway = {
enabled = false
}
gatewayClass = {
enabled = false
}
providers = {
kubernetesCRD = {
enabled = true
}
kubernetesIngress = {
enabled = true
}
}
ports = {
web = {
port = 80
exposedPort = 80
}
websecure = {
port = 443
exposedPort = 443
tls = {
enabled = true
}
}
}
service = {
type = "LoadBalancer"
annotations = {
"metallb.universe.tf/address-pool" = var.metallb.pool_name
}
spec = {
externalTrafficPolicy = "Local"
loadBalancerIP = var.traefik.load_balancer_ip
}
}
logs = {
general = {
level = "INFO"
}
}
resources = {
requests = {
cpu = "50m"
memory = "96Mi"
}
limits = {
memory = "256Mi"
}
}
})
]
}
resource "helm_release" "openebs" {
depends_on = [null_resource.calico_ready]
name = "openebs"

24
bootstrap/platform/variables.tf
View File

@ -124,16 +124,36 @@ variable "metallb" {
})
default = {
enabled = false
enabled = true
repository = "https://metallb.github.io/metallb"
version = "0.16.0"
namespace = "metallb-system"
address_pool = []
address_pool = ["192.168.100.240-192.168.100.240"]
l2_advertisement_enabled = true
pool_name = "homelab-lan"
}
}
variable "traefik" {
type = object({
enabled = bool
repository = string
chart_version = string
namespace = string
load_balancer_ip = string
ingress_class = string
})
default = {
enabled = true
repository = "https://helm.traefik.io/traefik"
chart_version = "40.2.0"
namespace = "traefik"
load_balancer_ip = "192.168.100.240"
ingress_class = "traefik"
}
}
variable "observability" {
type = object({
namespace = string

2
lab.sh
View File

@ -122,6 +122,8 @@ adopt_platform_existing_resources() {
adopt_tofu_helm_release "${stack}" "helm_release.calico_crds" "tigera-operator" "calico-crds"
adopt_tofu_helm_release "${stack}" "helm_release.calico" "tigera-operator" "calico"
adopt_tofu_helm_release "${stack}" "helm_release.openebs" "openebs" "openebs"
adopt_tofu_helm_release "${stack}" "helm_release.metallb[\"enabled\"]" "metallb-system" "metallb"
adopt_tofu_helm_release "${stack}" "helm_release.traefik[\"enabled\"]" "traefik" "traefik"
adopt_tofu_helm_release "${stack}" "helm_release.argocd" "argocd" "argocd"
adopt_tofu_helm_release "${stack}" "helm_release.kyverno" "kyverno" "kyverno"
adopt_tofu_helm_release "${stack}" "helm_release.kyverno_policies" "kyverno" "kyverno-policies"