Add signed SBOM enforcement for local images

This commit is contained in:
juvdiaz 2026-06-04 12:12:42 -06:00
parent a5fc43b78e
commit d810fc61c0
9 changed files with 411 additions and 12 deletions

View File

@ -57,7 +57,8 @@ accidentally modify the cluster.
- installs OpenEBS - installs OpenEBS
- creates `openebs-hostpath-retain` - creates `openebs-hostpath-retain`
- installs Argo CD - installs Argo CD
- installs Kyverno with audit-first baseline Pod Security policies - installs Kyverno with audit-first baseline Pod Security policies and
homelab image signature/SBOM admission controls
- registers the private GitOps repo without storing the SSH private key in - registers the private GitOps repo without storing the SSH private key in
Terraform state Terraform state
@ -69,7 +70,7 @@ accidentally modify the cluster.
listens on loopback inside the container, OPcache is enabled, and listens on loopback inside the container, OPcache is enabled, and
pre-rendered HTML handles the read-heavy pages pre-rendered HTML handles the read-heavy pages
- default apps are `container-registry`, `website-production`, - default apps are `container-registry`, `website-production`,
`demos-static`, and `heimdall` `demos-static`, `heimdall`, and `supply-chain-policy`
5. `bootstrap/edge` 5. `bootstrap/edge`
- connects to the OCI jump box - connects to the OCI jump box
@ -332,6 +333,24 @@ settings without blocking existing pods during the first rollout. After reports
are clean, individual policies can be promoted to `Enforce` in are clean, individual policies can be promoted to `Enforce` in
`bootstrap/platform/main.tf`. `bootstrap/platform/main.tf`.
`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for
the images built by this repo and pushed to the local registry. The policy
targets `192.168.100.68:30500/php-website:*` and
`192.168.100.68:30500/demos-static:*`, mutates admitted pods to image digests,
and denies pods unless the image has both:
- a valid Cosign signature from the homelab signing key
- a signed SPDX SBOM attestation
The private Cosign key, generated password, downloaded Cosign binary, and image
state files live under `.lab/`, which is ignored by Git. `./lab.sh apps` creates
or reuses `.lab/cosign.key`, publishes `.lab/cosign.pub` into the Kyverno
namespace as `homelab-cosign-public-key`, builds images with BuildKit SBOM
attestations, signs the pushed images, attaches a signed SPDX SBOM predicate, and
verifies both artifacts before refreshing the workload Applications. Set
`COSIGN_PASSWORD` if you want to manage the key password externally; otherwise
the Debian runner creates `.lab/cosign.password` for non-interactive runs.
## DNS Cache ## DNS Cache
`bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with `bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with
@ -616,13 +635,15 @@ them through the edge path at `/demo-apps/`.
- a content-hash `php-website` tag generated by `lab.sh` and passed to Argo CD - a content-hash `php-website` tag generated by `lab.sh` and passed to Argo CD
as a Kustomize image override as a Kustomize image override
- `demos-static:latest` from `apps/demos-static` - `demos-static:latest` from `apps/demos-static`
- Cosign signatures and signed SPDX SBOM attestations for both pushed images
The website manifest keeps the stable base image name `php-website:bootstrap`. The website manifest keeps the stable base image name `php-website:bootstrap`.
During bootstrap, `lab.sh` hashes `apps/website`, builds During bootstrap, `lab.sh` hashes `apps/website`, builds
`<registry>/php-website:src-<hash>`, exports that exact reference through `<registry>/php-website:src-<hash>`, exports that exact reference through
`TF_VAR_website_image_ref`, and the Argo CD Application applies it through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through
Kustomize. This keeps the GitOps source generic while the deployed image remains Kustomize. This keeps the GitOps source generic while the deployed image remains
immutable. immutable. The Kyverno supply-chain policy mutates admitted pods to the verified
image digest, so the workload runs the same digest that was signed and attested.
After `./lab.sh apps`, the live deployment image should be a content-hash tag, After `./lab.sh apps`, the live deployment image should be a content-hash tag,
for example `192.168.100.68:30500/php-website:src-...`. If it still shows for example `192.168.100.68:30500/php-website:src-...`. If it still shows
@ -667,7 +688,8 @@ Current demo inventory:
The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base.
Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by
the app image build. the app image build and with the image globs in
`apps/supply-chain-policy/local-registry-image-policy.yaml`.
Keep the `.terraform.lock.hcl` files committed. They pin provider selections and Keep the `.terraform.lock.hcl` files committed. They pin provider selections and
make bootstrap behavior reproducible across nodes and rebuilds. make bootstrap behavior reproducible across nodes and rebuilds.

View File

@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kyverno-cosign-key-rbac.yaml
- local-registry-image-policy.yaml

View File

@ -0,0 +1,31 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: homelab-cosign-public-key-reader
namespace: kyverno
rules:
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
- homelab-cosign-public-key
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: homelab-cosign-public-key-reader
namespace: kyverno
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: homelab-cosign-public-key-reader
subjects:
- kind: ServiceAccount
name: kyverno-admission-controller
namespace: kyverno
- kind: ServiceAccount
name: kyverno-background-controller
namespace: kyverno

View File

@ -0,0 +1,68 @@
apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
name: homelab-local-registry-supply-chain
annotations:
policies.kyverno.io/title: Verify Homelab Local Registry Images
policies.kyverno.io/category: Software Supply Chain Security
policies.kyverno.io/severity: high
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Requires homelab-built local registry images to have a valid Cosign
signature and a signed SPDX SBOM attestation before pods are admitted.
spec:
failurePolicy: Fail
validationActions:
- Deny
webhookConfiguration:
timeoutSeconds: 15
evaluation:
background:
enabled: false
matchConstraints:
resourceRules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- pods
matchImageReferences:
- glob: "192.168.100.68:30500/php-website:*"
- glob: "192.168.100.68:30500/demos-static:*"
validationConfigurations:
mutateDigest: true
required: true
verifyDigest: true
credentials:
allowInsecureRegistry: true
variables:
- name: homelabSigningKey
expression: 'resource.Get("v1", "configmaps", "kyverno", "homelab-cosign-public-key")'
attestors:
- name: homelabCosign
cosign:
key:
expression: variables.homelabSigningKey.data["cosign.pub"]
ctlog:
insecureIgnoreTlog: true
attestations:
- name: spdxSbom
intoto:
type: https://spdx.dev/Document
validations:
- expression: >-
images.containers.map(image, verifyImageSignatures(image,
[attestors.homelabCosign])).all(result, result > 0)
message: local registry image must be signed by the homelab Cosign key
- expression: >-
images.containers.map(image, verifyAttestationSignatures(image,
attestations.spdxSbom, [attestors.homelabCosign])).all(result, result > 0)
message: local registry image must include a signed SPDX SBOM attestation
- expression: >-
images.containers.map(image, extractPayload(image,
attestations.spdxSbom).SPDXID == "SPDXRef-DOCUMENT").all(result, result)
message: local registry image SBOM attestation must contain an SPDX document

View File

@ -17,6 +17,7 @@ $activityKeys = [
'blog_activity_13', 'blog_activity_13',
'blog_activity_14', 'blog_activity_14',
'blog_activity_15', 'blog_activity_15',
'blog_activity_16',
]; ];
$todoKeys = [ $todoKeys = [
@ -55,6 +56,7 @@ $stackKeys = [
'blog_stack_15', 'blog_stack_15',
'blog_stack_16', 'blog_stack_16',
'blog_stack_17', 'blog_stack_17',
'blog_stack_18',
]; ];
$caseStudyCards = [ $caseStudyCards = [
@ -167,6 +169,11 @@ $stackSourceLinks = [
['label' => 'Apache/FPM config', 'path' => 'apps/website/httpd-php-fpm.conf'], ['label' => 'Apache/FPM config', 'path' => 'apps/website/httpd-php-fpm.conf'],
['label' => 'edge nginx template', 'path' => 'bootstrap/edge/templates/default.conf.tftpl'], ['label' => 'edge nginx template', 'path' => 'bootstrap/edge/templates/default.conf.tftpl'],
], ],
'blog_stack_18' => [
['label' => 'supply-chain policy', 'path' => 'apps/supply-chain-policy/local-registry-image-policy.yaml'],
['label' => 'Cosign build flow', 'path' => 'lab.sh'],
['label' => 'apps variables', 'path' => 'bootstrap/apps/variables.tf'],
],
]; ];
function renderStackSourceLinks(string $stackKey, array $sourceLinks, string $sourceBase): void { function renderStackSourceLinks(string $stackKey, array $sourceLinks, string $sourceBase): void {

View File

@ -149,6 +149,7 @@ return [
'blog_stack_15' => 'OpenWrt is handled separately from the Debian golden-node template. The lab downloads the upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, and keeps it disabled unless LAB_OPENWRT_VM=true is set.', 'blog_stack_15' => 'OpenWrt is handled separately from the Debian golden-node template. The lab downloads the upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, and keeps it disabled unless LAB_OPENWRT_VM=true is set.',
'blog_stack_16' => 'The monitoring layer now includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, and kube-state-metrics, with placement guardrails so platform add-ons stop defaulting to the control plane.', 'blog_stack_16' => 'The monitoring layer now includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, and kube-state-metrics, with placement guardrails so platform add-ons stop defaulting to the control plane.',
'blog_stack_17' => 'The portfolio website keeps Apache at the core, runs PHP through PHP-FPM with OPcache, pre-renders read-heavy pages into static HTML during the image build, and leaves PHP for write and translation endpoints.', 'blog_stack_17' => 'The portfolio website keeps Apache at the core, runs PHP through PHP-FPM with OPcache, pre-renders read-heavy pages into static HTML during the image build, and leaves PHP for write and translation endpoints.',
'blog_stack_18' => 'The supply-chain path now uses BuildKit SBOM attestations, Cosign signatures, a cluster-published public key, and a Kyverno ImageValidatingPolicy that denies unsigned local-registry pods.',
'blog_arch_kicker' => 'Architecture map', 'blog_arch_kicker' => 'Architecture map',
'blog_arch_title' => 'The homelab, end to end', 'blog_arch_title' => 'The homelab, end to end',
'blog_arch_intro' => 'The current delivery path starts with a push to Gitea, runs local validation, builds arm64 images, syncs the validated commit into the GitOps mirror, and lets Argo CD reconcile from the app workers. The infrastructure path stays manual through lab.sh, including the PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, and the opt-in OpenWrt firewall VM, while the OCI edge routes public traffic back through the private path.', 'blog_arch_intro' => 'The current delivery path starts with a push to Gitea, runs local validation, builds arm64 images, syncs the validated commit into the GitOps mirror, and lets Argo CD reconcile from the app workers. The infrastructure path stays manual through lab.sh, including the PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, and the opt-in OpenWrt firewall VM, while the OCI edge routes public traffic back through the private path.',
@ -172,6 +173,7 @@ return [
'blog_activity_13' => 'Installed the monitoring stack and moved platform add-ons such as Argo CD, Kyverno, and prometheus-stack work toward app-worker placement instead of treating the control plane as spare capacity.', 'blog_activity_13' => 'Installed the monitoring stack and moved platform add-ons such as Argo CD, Kyverno, and prometheus-stack work toward app-worker placement instead of treating the control plane as spare capacity.',
'blog_activity_14' => 'Reworked the website image to keep Apache as the web server while moving PHP execution to PHP-FPM, enabling OPcache, and validating the runtime through Gitea Actions instead of local image builds.', 'blog_activity_14' => 'Reworked the website image to keep Apache as the web server while moving PHP execution to PHP-FPM, enabling OPcache, and validating the runtime through Gitea Actions instead of local image builds.',
'blog_activity_15' => 'Pre-rendered the read-heavy website pages into static HTML, moved visitor ideas behind a small JSON endpoint, added cache headers, enabled HTTP/2 and connection reuse at the OCI edge, and measured reused HTTP/2 asset loads around 60ms.', 'blog_activity_15' => 'Pre-rendered the read-heavy website pages into static HTML, moved visitor ideas behind a small JSON endpoint, added cache headers, enabled HTTP/2 and connection reuse at the OCI edge, and measured reused HTTP/2 asset loads around 60ms.',
'blog_activity_16' => 'Added image signing and SBOM enforcement: lab.sh now builds with SBOM metadata, signs php-website and demos-static with Cosign, attaches signed SPDX SBOM attestations, and Kyverno denies unsigned local-registry pods.',
'blog_todo_kicker' => 'Improvement backlog', 'blog_todo_kicker' => 'Improvement backlog',
'blog_todo_title' => 'Todo list for the next homelab pass', 'blog_todo_title' => 'Todo list for the next homelab pass',
'blog_todo_intro' => 'These are improvement proposals, not chores for the sake of chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, or makes the platform easier to operate when something fails.', 'blog_todo_intro' => 'These are improvement proposals, not chores for the sake of chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, or makes the platform easier to operate when something fails.',
@ -179,7 +181,7 @@ return [
'blog_todo_2' => 'Keep the Debian bare GitOps mirror as the cluster source and add object-storage backups when OCI storage is ready.', 'blog_todo_2' => 'Keep the Debian bare GitOps mirror as the cluster source and add object-storage backups when OCI storage is ready.',
'blog_todo_3' => 'Add a real OpenTofu remote state backend with backup, locking, and a documented recovery path.', 'blog_todo_3' => 'Add a real OpenTofu remote state backend with backup, locking, and a documented recovery path.',
'blog_todo_4' => 'Replace the remaining mutable latest image references with immutable tags or digest pins for demo workloads; the website image now uses a content-hash tag.', 'blog_todo_4' => 'Replace the remaining mutable latest image references with immutable tags or digest pins for demo workloads; the website image now uses a content-hash tag.',
'blog_todo_5' => 'Generate SBOMs and sign images so the local registry can prove what it is serving.', 'blog_todo_5' => 'Move the Cosign private key into Vault or another secret manager so the runner no longer owns long-lived signing material on disk.',
'blog_todo_6' => 'Add Renovate or Dependabot-style dependency updates for base images, Helm charts, and GitHub/Gitea Actions.', 'blog_todo_6' => 'Add Renovate or Dependabot-style dependency updates for base images, Helm charts, and GitHub/Gitea Actions.',
'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, and documented exceptions for platform components.', 'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, and documented exceptions for platform components.',
'blog_todo_8' => 'Turn the installed observability stack into useful operations views: a few high-signal dashboards, alerts for node health, storage pressure, certificate expiry, and failed app syncs.', 'blog_todo_8' => 'Turn the installed observability stack into useful operations views: a few high-signal dashboards, alerts for node health, storage pressure, certificate expiry, and failed app syncs.',

View File

@ -149,6 +149,7 @@ return [
'blog_stack_15' => 'OpenWrt handled separately from Debian golden-node template. Lab downloads upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, ihuan keeps disabled unless LAB_OPENWRT_VM=true.', 'blog_stack_15' => 'OpenWrt handled separately from Debian golden-node template. Lab downloads upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, ihuan keeps disabled unless LAB_OPENWRT_VM=true.',
'blog_stack_16' => 'Monitoring layer axcan includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, ihuan kube-state-metrics, ika placement guardrails para platform add-ons ma amo default to control plane.', 'blog_stack_16' => 'Monitoring layer axcan includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, ihuan kube-state-metrics, ika placement guardrails para platform add-ons ma amo default to control plane.',
'blog_stack_17' => 'Portfolio website keeps Apache at core, runs PHP through PHP-FPM ika OPcache, pre-renders read-heavy pages into static HTML during image build, ihuan leaves PHP para write ihuan translation endpoints.', 'blog_stack_17' => 'Portfolio website keeps Apache at core, runs PHP through PHP-FPM ika OPcache, pre-renders read-heavy pages into static HTML during image build, ihuan leaves PHP para write ihuan translation endpoints.',
'blog_stack_18' => 'Supply-chain path axcan uses BuildKit SBOM attestations, Cosign signatures, public key ipan cluster, ihuan Kyverno ImageValidatingPolicy in quidenies unsigned local-registry pods.',
'blog_arch_kicker' => 'Architecture mapa', 'blog_arch_kicker' => 'Architecture mapa',
'blog_arch_title' => 'Homelab, end to end', 'blog_arch_title' => 'Homelab, end to end',
'blog_arch_intro' => 'Current delivery path starts ika push to Gitea, runs local validation, builds arm64 images, syncs validated commit into GitOps mirror, ihuan lets Argo CD reconcile from app workers. Infrastructure path stays manual through lab.sh, including PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, ihuan opt-in OpenWrt firewall VM, while OCI edge routes public traffic back through private path.', 'blog_arch_intro' => 'Current delivery path starts ika push to Gitea, runs local validation, builds arm64 images, syncs validated commit into GitOps mirror, ihuan lets Argo CD reconcile from app workers. Infrastructure path stays manual through lab.sh, including PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, ihuan opt-in OpenWrt firewall VM, while OCI edge routes public traffic back through private path.',
@ -172,6 +173,7 @@ return [
'blog_activity_13' => 'Installed monitoring stack ihuan moved platform add-ons such as Argo CD, Kyverno, ihuan prometheus-stack toward app-worker placement instead of treating control plane as spare capacity.', 'blog_activity_13' => 'Installed monitoring stack ihuan moved platform add-ons such as Argo CD, Kyverno, ihuan prometheus-stack toward app-worker placement instead of treating control plane as spare capacity.',
'blog_activity_14' => 'Reworked website image para Apache ma mocahua web server while PHP runs through PHP-FPM, OPcache enabled, ihuan runtime validated through Gitea Actions instead of local image builds.', 'blog_activity_14' => 'Reworked website image para Apache ma mocahua web server while PHP runs through PHP-FPM, OPcache enabled, ihuan runtime validated through Gitea Actions instead of local image builds.',
'blog_activity_15' => 'Pre-rendered read-heavy website pages into static HTML, moved visitor ideas behind small JSON endpoint, added cache headers, enabled HTTP/2 ihuan connection reuse at OCI edge, ihuan measured reused HTTP/2 asset loads around 60ms.', 'blog_activity_15' => 'Pre-rendered read-heavy website pages into static HTML, moved visitor ideas behind small JSON endpoint, added cache headers, enabled HTTP/2 ihuan connection reuse at OCI edge, ihuan measured reused HTTP/2 asset loads around 60ms.',
'blog_activity_16' => 'Added image signing ihuan SBOM enforcement: lab.sh axcan builds ika SBOM metadata, signs php-website ihuan demos-static ika Cosign, attaches signed SPDX SBOM attestations, ihuan Kyverno quidenies unsigned local-registry pods.',
'blog_todo_kicker' => 'Improvement tlatecpanaliztli', 'blog_todo_kicker' => 'Improvement tlatecpanaliztli',
'blog_todo_title' => 'Todo list para next homelab pass', 'blog_todo_title' => 'Todo list para next homelab pass',
'blog_todo_intro' => 'Inin cateh improvement proposals, amo chores zan pampa chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, o makes platform easier to operate quema tlein fails.', 'blog_todo_intro' => 'Inin cateh improvement proposals, amo chores zan pampa chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, o makes platform easier to operate quema tlein fails.',
@ -179,7 +181,7 @@ return [
'blog_todo_2' => 'Keep Debian bare GitOps mirror as cluster source ihuan add object-storage backups quema OCI storage ready.', 'blog_todo_2' => 'Keep Debian bare GitOps mirror as cluster source ihuan add object-storage backups quema OCI storage ready.',
'blog_todo_3' => 'Add real OpenTofu remote state backend ika backup, locking, ihuan documented recovery path.', 'blog_todo_3' => 'Add real OpenTofu remote state backend ika backup, locking, ihuan documented recovery path.',
'blog_todo_4' => 'Replace remaining mutable latest image references ika immutable tags o digest pins para demo workloads; website image axcan uses content-hash tag.', 'blog_todo_4' => 'Replace remaining mutable latest image references ika immutable tags o digest pins para demo workloads; website image axcan uses content-hash tag.',
'blog_todo_5' => 'Generate SBOMs ihuan sign images so local registry can prove tlein serving.', 'blog_todo_5' => 'Move Cosign private key into Vault o occe secret manager so runner ayacmo owns long-lived signing material on disk.',
'blog_todo_6' => 'Add Renovate o Dependabot-style dependency updates para base images, Helm charts, ihuan GitHub/Gitea Actions.', 'blog_todo_6' => 'Add Renovate o Dependabot-style dependency updates para base images, Helm charts, ihuan GitHub/Gitea Actions.',
'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, ihuan documented exceptions para platform components.', 'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, ihuan documented exceptions para platform components.',
'blog_todo_8' => 'Turn installed observability stack into useful operations views: few high-signal dashboards, alerts para node health, storage pressure, certificate expiry, ihuan failed app syncs.', 'blog_todo_8' => 'Turn installed observability stack into useful operations views: few high-signal dashboards, alerts para node health, storage pressure, certificate expiry, ihuan failed app syncs.',

View File

@ -66,5 +66,14 @@ variable "applications" {
self_heal = true self_heal = true
create_namespace = true create_namespace = true
} }
supply-chain-policy = {
project = "default"
path = "apps/supply-chain-policy"
namespace = "kyverno"
target_revision = "main"
prune = true
self_heal = true
create_namespace = false
}
} }
} }

265
lab.sh
View File

@ -4,6 +4,14 @@ set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BUILDX_CONFIG="/tmp/buildx-config.toml" BUILDX_CONFIG="/tmp/buildx-config.toml"
KUBECONFIG_PATH="${KUBECONFIG_PATH:-${TF_VAR_kubeconfig_path:-/home/jv/.kube/config}}" KUBECONFIG_PATH="${KUBECONFIG_PATH:-${TF_VAR_kubeconfig_path:-/home/jv/.kube/config}}"
COSIGN_KEY_PREFIX="${COSIGN_KEY_PREFIX:-${REPO_ROOT}/.lab/cosign}"
COSIGN_KEY_PATH="${COSIGN_KEY_PATH:-${COSIGN_KEY_PREFIX}.key}"
COSIGN_PUBLIC_KEY_PATH="${COSIGN_PUBLIC_KEY_PATH:-${COSIGN_KEY_PREFIX}.pub}"
COSIGN_PASSWORD_FILE="${COSIGN_PASSWORD_FILE:-${REPO_ROOT}/.lab/cosign.password}"
COSIGN_VERSION="${COSIGN_VERSION:-2.6.3}"
COSIGN_BIN="${COSIGN_BIN:-}"
HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP="${HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP:-homelab-cosign-public-key}"
HOMELAB_SBOM_PREDICATE_TYPE="${HOMELAB_SBOM_PREDICATE_TYPE:-https://spdx.dev/Document}"
trap 'rm -f "${BUILDX_CONFIG}"' EXIT trap 'rm -f "${BUILDX_CONFIG}"' EXIT
@ -182,6 +190,14 @@ adopt_apps_existing_resources() {
"argoproj.io/v1alpha1" \ "argoproj.io/v1alpha1" \
"Application" \ "Application" \
"heimdall" "heimdall"
adopt_tofu_kubernetes_manifest \
"${stack}" \
'kubernetes_manifest.argocd_application["supply-chain-policy"]' \
"${namespace}" \
"applications.argoproj.io" \
"argoproj.io/v1alpha1" \
"Application" \
"supply-chain-policy"
} }
ensure_homelab_node_labels() { ensure_homelab_node_labels() {
@ -1613,6 +1629,218 @@ EOF
docker buildx inspect --bootstrap docker buildx inspect --bootstrap
} }
ensure_cosign_available() {
local arch
local cosign_url
local target_dir
local target_path
local tmp_path
if [[ -n "${COSIGN_BIN}" ]]; then
if [[ -x "${COSIGN_BIN}" ]]; then
return 0
fi
echo "COSIGN_BIN points to ${COSIGN_BIN}, but it is not executable." >&2
exit 1
fi
if command -v cosign >/dev/null 2>&1; then
COSIGN_BIN="$(command -v cosign)"
return 0
fi
case "$(uname -m)" in
x86_64 | amd64)
arch="amd64"
;;
aarch64 | arm64)
arch="arm64"
;;
*)
echo "Unsupported Cosign install architecture: $(uname -m)" >&2
exit 1
;;
esac
target_dir="${REPO_ROOT}/.lab/bin"
target_path="${target_dir}/cosign-v${COSIGN_VERSION}-linux-${arch}"
if [[ -x "${target_path}" ]]; then
COSIGN_BIN="${target_path}"
return 0
fi
if ! command -v curl >/dev/null 2>&1; then
echo "curl is required to download Cosign ${COSIGN_VERSION}." >&2
exit 1
fi
mkdir -p "${target_dir}"
tmp_path="${target_path}.tmp"
cosign_url="https://github.com/sigstore/cosign/releases/download/v${COSIGN_VERSION}/cosign-linux-${arch}"
echo "Installing Cosign ${COSIGN_VERSION} into ${target_path}..."
curl -fsSL "${cosign_url}" -o "${tmp_path}"
chmod 0755 "${tmp_path}"
mv "${tmp_path}" "${target_path}"
COSIGN_BIN="${target_path}"
}
ensure_cosign_password() {
if [[ -n "${COSIGN_PASSWORD:-}" ]]; then
return 0
fi
if [[ -r "${COSIGN_PASSWORD_FILE}" ]]; then
COSIGN_PASSWORD="$(<"${COSIGN_PASSWORD_FILE}")"
export COSIGN_PASSWORD
return 0
fi
mkdir -p "$(dirname "${COSIGN_PASSWORD_FILE}")"
if command -v openssl >/dev/null 2>&1; then
openssl rand -hex 32 > "${COSIGN_PASSWORD_FILE}"
elif command -v python3 >/dev/null 2>&1; then
python3 - <<'PY' > "${COSIGN_PASSWORD_FILE}"
import secrets
print(secrets.token_hex(32))
PY
else
echo "openssl or python3 is required to generate a non-interactive Cosign key password." >&2
exit 1
fi
chmod 0600 "${COSIGN_PASSWORD_FILE}"
COSIGN_PASSWORD="$(<"${COSIGN_PASSWORD_FILE}")"
export COSIGN_PASSWORD
}
run_cosign() {
ensure_cosign_available
ensure_cosign_password
COSIGN_PASSWORD="${COSIGN_PASSWORD}" "${COSIGN_BIN}" "$@"
}
ensure_cosign_keypair() {
ensure_cosign_available
ensure_cosign_password
if [[ -f "${COSIGN_KEY_PATH}" && -f "${COSIGN_PUBLIC_KEY_PATH}" ]]; then
return 0
fi
if [[ -f "${COSIGN_KEY_PATH}" || -f "${COSIGN_PUBLIC_KEY_PATH}" ]]; then
echo "Found only one Cosign key file. Expected both ${COSIGN_KEY_PATH} and ${COSIGN_PUBLIC_KEY_PATH}." >&2
exit 1
fi
mkdir -p "$(dirname "${COSIGN_KEY_PATH}")"
echo "Generating homelab Cosign key pair under ${REPO_ROOT}/.lab..."
COSIGN_PASSWORD="${COSIGN_PASSWORD}" "${COSIGN_BIN}" generate-key-pair --output-key-prefix "${COSIGN_KEY_PREFIX}"
chmod 0600 "${COSIGN_KEY_PATH}"
chmod 0644 "${COSIGN_PUBLIC_KEY_PATH}"
}
publish_cosign_public_key_config_map() {
kubectl --kubeconfig "${KUBECONFIG}" create namespace kyverno --dry-run=client -o yaml |
kubectl --kubeconfig "${KUBECONFIG}" apply -f -
kubectl --kubeconfig "${KUBECONFIG}" -n kyverno create configmap "${HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP}" \
--from-file=cosign.pub="${COSIGN_PUBLIC_KEY_PATH}" \
--dry-run=client -o yaml |
kubectl --kubeconfig "${KUBECONFIG}" apply -f -
}
cosign_registry_flags() {
local registry_endpoint="$1"
local image_ref="$2"
if [[ "${image_ref}" == "${registry_endpoint}/"* ]]; then
printf '%s\n' "--allow-http-registry"
fi
}
image_supply_chain_metadata_exists() {
local image_ref="$1"
local registry_endpoint="$2"
local -a registry_flags=()
mapfile -t registry_flags < <(cosign_registry_flags "${registry_endpoint}" "${image_ref}")
run_cosign verify \
--key "${COSIGN_PUBLIC_KEY_PATH}" \
--insecure-ignore-tlog=true \
"${registry_flags[@]}" \
"${image_ref}" >/dev/null 2>&1 &&
run_cosign verify-attestation \
--key "${COSIGN_PUBLIC_KEY_PATH}" \
--type "${HOMELAB_SBOM_PREDICATE_TYPE}" \
--insecure-ignore-tlog=true \
"${registry_flags[@]}" \
"${image_ref}" >/dev/null 2>&1
}
write_image_sbom_predicate() {
local image_ref="$1"
local output_file="$2"
docker buildx imagetools inspect "${image_ref}" --format '{{ json .SBOM.SPDX }}' > "${output_file}"
python3 - "${output_file}" <<'PY'
import json
import sys
path = sys.argv[1]
with open(path, encoding="utf-8") as handle:
document = json.load(handle)
if not isinstance(document, dict):
raise SystemExit("image SBOM predicate is not a JSON object")
if document.get("SPDXID") != "SPDXRef-DOCUMENT":
raise SystemExit("image SBOM predicate is not an SPDX document")
if not document.get("packages"):
raise SystemExit("image SBOM predicate does not include packages")
PY
}
publish_image_supply_chain_metadata() {
local image_ref="$1"
local registry_endpoint="$2"
local source_hash="$3"
local sbom_file
local -a registry_flags=()
if image_supply_chain_metadata_exists "${image_ref}" "${registry_endpoint}"; then
echo "Image ${image_ref} already has a valid Cosign signature and signed SPDX SBOM."
return 0
fi
mapfile -t registry_flags < <(cosign_registry_flags "${registry_endpoint}" "${image_ref}")
sbom_file="$(mktemp)"
write_image_sbom_predicate "${image_ref}" "${sbom_file}"
echo "Signing image ${image_ref} and attaching signed SPDX SBOM..."
run_cosign sign \
--yes \
--key "${COSIGN_KEY_PATH}" \
--tlog-upload=false \
-a "homelab.dev/source-hash=${source_hash}" \
"${registry_flags[@]}" \
"${image_ref}"
run_cosign attest \
--yes \
--key "${COSIGN_KEY_PATH}" \
--predicate "${sbom_file}" \
--type "${HOMELAB_SBOM_PREDICATE_TYPE}" \
--tlog-upload=false \
"${registry_flags[@]}" \
"${image_ref}"
rm -f "${sbom_file}"
if ! image_supply_chain_metadata_exists "${image_ref}" "${registry_endpoint}"; then
echo "Cosign verification failed after signing ${image_ref}." >&2
exit 1
fi
}
dump_argocd_debug() { dump_argocd_debug() {
local app="$1" local app="$1"
@ -1669,6 +1897,24 @@ wait_for_namespaced_resource() {
done done
} }
wait_for_cluster_resource() {
local kind="$1"
local name="$2"
local app="$3"
local timeout_seconds="$4"
local elapsed=0
until kubectl --kubeconfig "${KUBECONFIG}" get "${kind}/${name}" >/dev/null 2>&1; do
if ((elapsed >= timeout_seconds)); then
echo "Timed out waiting for ${kind}/${name} from Argo CD app ${app}" >&2
dump_argocd_debug "${app}"
exit 1
fi
sleep 5
elapsed=$((elapsed + 5))
done
}
wait_for_deployment_ready() { wait_for_deployment_ready() {
local namespace="$1" local namespace="$1"
local deployment="$2" local deployment="$2"
@ -2595,6 +2841,9 @@ apps() {
export TF_VAR_kubeconfig_path="${TF_VAR_kubeconfig_path:-${KUBECONFIG_PATH}}" export TF_VAR_kubeconfig_path="${TF_VAR_kubeconfig_path:-${KUBECONFIG_PATH}}"
export KUBECONFIG="${TF_VAR_kubeconfig_path}" export KUBECONFIG="${TF_VAR_kubeconfig_path}"
ensure_cosign_keypair
publish_cosign_public_key_config_map
if [[ "${TF_VAR_registry_endpoint}" != "${registry_endpoint}" ]]; then if [[ "${TF_VAR_registry_endpoint}" != "${registry_endpoint}" ]]; then
echo "TF_VAR_registry_endpoint changed after registry endpoint resolution (${registry_endpoint})" >&2 echo "TF_VAR_registry_endpoint changed after registry endpoint resolution (${registry_endpoint})" >&2
exit 1 exit 1
@ -2621,14 +2870,15 @@ apps() {
run_tofu_stack "bootstrap/apps" run_tofu_stack "bootstrap/apps"
refresh_argocd_application container-registry refresh_argocd_application container-registry
refresh_argocd_application demos-static refresh_argocd_application supply-chain-policy
refresh_argocd_application website-production
wait_for_cluster_resource imagevalidatingpolicy.policies.kyverno.io homelab-local-registry-supply-chain supply-chain-policy 300
wait_for_namespace container-registry container-registry 300 wait_for_namespace container-registry container-registry 300
wait_for_namespaced_resource container-registry deployment local-registry container-registry 300 wait_for_namespaced_resource container-registry deployment local-registry container-registry 300
wait_for_deployment_ready container-registry local-registry container-registry 300 wait_for_deployment_ready container-registry local-registry container-registry 300
if website_image_is_current "${website_image_state_file}" "${website_source_hash}" "${website_platforms}" "${website_image_ref}" "${registry_endpoint}"; then if website_image_is_current "${website_image_state_file}" "${website_source_hash}" "${website_platforms}" "${website_image_ref}" "${registry_endpoint}" &&
image_supply_chain_metadata_exists "${website_image_ref}" "${registry_endpoint}"; then
echo "Website image ${website_image_ref} is already current (${website_source_hash}); skipping build." echo "Website image ${website_image_ref} is already current (${website_source_hash}); skipping build."
else else
echo "Building website image ${website_image_ref} for ${website_platforms} (${website_source_hash})..." echo "Building website image ${website_image_ref} for ${website_platforms} (${website_source_hash})..."
@ -2642,7 +2892,7 @@ apps() {
--network host \ --network host \
--platform "${website_platforms}" \ --platform "${website_platforms}" \
--provenance=false \ --provenance=false \
--sbom=false \ --sbom=true \
--label "dev.homelab.website.source-hash=${website_source_hash}" \ --label "dev.homelab.website.source-hash=${website_source_hash}" \
-t "${website_image_ref}" \ -t "${website_image_ref}" \
-f "${REPO_ROOT}/apps/website/Dockerfile" \ -f "${REPO_ROOT}/apps/website/Dockerfile" \
@ -2650,8 +2900,10 @@ apps() {
--push --push
website_image_built=true website_image_built=true
fi fi
publish_image_supply_chain_metadata "${website_image_ref}" "${registry_endpoint}" "${website_source_hash}"
if demos_image_is_current "${demos_image_state_file}" "${demos_source_hash}" "${demos_platforms}" "${demos_image_ref}" "${registry_endpoint}"; then if demos_image_is_current "${demos_image_state_file}" "${demos_source_hash}" "${demos_platforms}" "${demos_image_ref}" "${registry_endpoint}" &&
image_supply_chain_metadata_exists "${demos_image_ref}" "${registry_endpoint}"; then
echo "Demos image ${demos_image_ref} is already current (${demos_source_hash}); skipping build." echo "Demos image ${demos_image_ref} is already current (${demos_source_hash}); skipping build."
else else
echo "Building demos image ${demos_image_ref} for ${demos_platforms} (${demos_source_hash})..." echo "Building demos image ${demos_image_ref} for ${demos_platforms} (${demos_source_hash})..."
@ -2665,7 +2917,7 @@ apps() {
--network host \ --network host \
--platform "${demos_platforms}" \ --platform "${demos_platforms}" \
--provenance=false \ --provenance=false \
--sbom=false \ --sbom=true \
--label "dev.homelab.demos.source-hash=${demos_source_hash}" \ --label "dev.homelab.demos.source-hash=${demos_source_hash}" \
-t "${demos_image_ref}" \ -t "${demos_image_ref}" \
-f "${REPO_ROOT}/apps/demos-static/Dockerfile" \ -f "${REPO_ROOT}/apps/demos-static/Dockerfile" \
@ -2673,6 +2925,7 @@ apps() {
--push --push
demos_image_built=true demos_image_built=true
fi fi
publish_image_supply_chain_metadata "${demos_image_ref}" "${registry_endpoint}" "${demos_source_hash}"
refresh_argocd_application website-production refresh_argocd_application website-production
wait_for_namespace website-production website-production 300 wait_for_namespace website-production website-production 300