diff --git a/README.md b/README.md index 5050a06..745a83a 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,8 @@ accidentally modify the cluster. - installs OpenEBS - creates `openebs-hostpath-retain` - installs Argo CD - - installs Kyverno with audit-first baseline Pod Security policies + - installs Kyverno with audit-first baseline Pod Security policies and + homelab image signature/SBOM admission controls - registers the private GitOps repo without storing the SSH private key in Terraform state @@ -69,7 +70,7 @@ accidentally modify the cluster. listens on loopback inside the container, OPcache is enabled, and pre-rendered HTML handles the read-heavy pages - default apps are `container-registry`, `website-production`, - `demos-static`, and `heimdall` + `demos-static`, `heimdall`, and `supply-chain-policy` 5. `bootstrap/edge` - connects to the OCI jump box @@ -332,6 +333,24 @@ settings without blocking existing pods during the first rollout. After reports are clean, individual policies can be promoted to `Enforce` in `bootstrap/platform/main.tf`. +`apps/supply-chain-policy` adds a separate Kyverno `ImageValidatingPolicy` for +the images built by this repo and pushed to the local registry. The policy +targets `192.168.100.68:30500/php-website:*` and +`192.168.100.68:30500/demos-static:*`, mutates admitted pods to image digests, +and denies pods unless the image has both: + +- a valid Cosign signature from the homelab signing key +- a signed SPDX SBOM attestation + +The private Cosign key, generated password, downloaded Cosign binary, and image +state files live under `.lab/`, which is ignored by Git. `./lab.sh apps` creates +or reuses `.lab/cosign.key`, publishes `.lab/cosign.pub` into the Kyverno +namespace as `homelab-cosign-public-key`, builds images with BuildKit SBOM +attestations, signs the pushed images, attaches a signed SPDX SBOM predicate, and +verifies both artifacts before refreshing the workload Applications. Set +`COSIGN_PASSWORD` if you want to manage the key password externally; otherwise +the Debian runner creates `.lab/cosign.password` for non-interactive runs. + ## DNS Cache `bootstrap/platform` installs NodeLocal DNSCache in `kube-system` with @@ -616,13 +635,15 @@ them through the edge path at `/demo-apps/`. - a content-hash `php-website` tag generated by `lab.sh` and passed to Argo CD as a Kustomize image override - `demos-static:latest` from `apps/demos-static` +- Cosign signatures and signed SPDX SBOM attestations for both pushed images The website manifest keeps the stable base image name `php-website:bootstrap`. During bootstrap, `lab.sh` hashes `apps/website`, builds `/php-website:src-`, exports that exact reference through `TF_VAR_website_image_ref`, and the Argo CD Application applies it through Kustomize. This keeps the GitOps source generic while the deployed image remains -immutable. +immutable. The Kyverno supply-chain policy mutates admitted pods to the verified +image digest, so the workload runs the same digest that was signed and attested. After `./lab.sh apps`, the live deployment image should be a content-hash tag, for example `192.168.100.68:30500/php-website:src-...`. If it still shows @@ -667,7 +688,8 @@ Current demo inventory: The Kubernetes deployment uses `apps/website/web-app.yaml` as a Kustomize base. Keep `TF_VAR_registry_endpoint` aligned with the local registry endpoint used by -the app image build. +the app image build and with the image globs in +`apps/supply-chain-policy/local-registry-image-policy.yaml`. Keep the `.terraform.lock.hcl` files committed. They pin provider selections and make bootstrap behavior reproducible across nodes and rebuilds. diff --git a/apps/supply-chain-policy/kustomization.yaml b/apps/supply-chain-policy/kustomization.yaml new file mode 100644 index 0000000..b9fe210 --- /dev/null +++ b/apps/supply-chain-policy/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - kyverno-cosign-key-rbac.yaml + - local-registry-image-policy.yaml diff --git a/apps/supply-chain-policy/kyverno-cosign-key-rbac.yaml b/apps/supply-chain-policy/kyverno-cosign-key-rbac.yaml new file mode 100644 index 0000000..67542b5 --- /dev/null +++ b/apps/supply-chain-policy/kyverno-cosign-key-rbac.yaml @@ -0,0 +1,31 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: homelab-cosign-public-key-reader + namespace: kyverno +rules: + - apiGroups: + - "" + resources: + - configmaps + resourceNames: + - homelab-cosign-public-key + verbs: + - get +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: homelab-cosign-public-key-reader + namespace: kyverno +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: homelab-cosign-public-key-reader +subjects: + - kind: ServiceAccount + name: kyverno-admission-controller + namespace: kyverno + - kind: ServiceAccount + name: kyverno-background-controller + namespace: kyverno diff --git a/apps/supply-chain-policy/local-registry-image-policy.yaml b/apps/supply-chain-policy/local-registry-image-policy.yaml new file mode 100644 index 0000000..660ff9e --- /dev/null +++ b/apps/supply-chain-policy/local-registry-image-policy.yaml @@ -0,0 +1,68 @@ +apiVersion: policies.kyverno.io/v1 +kind: ImageValidatingPolicy +metadata: + name: homelab-local-registry-supply-chain + annotations: + policies.kyverno.io/title: Verify Homelab Local Registry Images + policies.kyverno.io/category: Software Supply Chain Security + policies.kyverno.io/severity: high + policies.kyverno.io/subject: Pod + policies.kyverno.io/description: >- + Requires homelab-built local registry images to have a valid Cosign + signature and a signed SPDX SBOM attestation before pods are admitted. +spec: + failurePolicy: Fail + validationActions: + - Deny + webhookConfiguration: + timeoutSeconds: 15 + evaluation: + background: + enabled: false + matchConstraints: + resourceRules: + - apiGroups: + - "" + apiVersions: + - v1 + operations: + - CREATE + - UPDATE + resources: + - pods + matchImageReferences: + - glob: "192.168.100.68:30500/php-website:*" + - glob: "192.168.100.68:30500/demos-static:*" + validationConfigurations: + mutateDigest: true + required: true + verifyDigest: true + credentials: + allowInsecureRegistry: true + variables: + - name: homelabSigningKey + expression: 'resource.Get("v1", "configmaps", "kyverno", "homelab-cosign-public-key")' + attestors: + - name: homelabCosign + cosign: + key: + expression: variables.homelabSigningKey.data["cosign.pub"] + ctlog: + insecureIgnoreTlog: true + attestations: + - name: spdxSbom + intoto: + type: https://spdx.dev/Document + validations: + - expression: >- + images.containers.map(image, verifyImageSignatures(image, + [attestors.homelabCosign])).all(result, result > 0) + message: local registry image must be signed by the homelab Cosign key + - expression: >- + images.containers.map(image, verifyAttestationSignatures(image, + attestations.spdxSbom, [attestors.homelabCosign])).all(result, result > 0) + message: local registry image must include a signed SPDX SBOM attestation + - expression: >- + images.containers.map(image, extractPayload(image, + attestations.spdxSbom).SPDXID == "SPDXRef-DOCUMENT").all(result, result) + message: local registry image SBOM attestation must contain an SPDX document diff --git a/apps/website/blog.php b/apps/website/blog.php index 0654caf..b79c407 100644 --- a/apps/website/blog.php +++ b/apps/website/blog.php @@ -17,6 +17,7 @@ $activityKeys = [ 'blog_activity_13', 'blog_activity_14', 'blog_activity_15', + 'blog_activity_16', ]; $todoKeys = [ @@ -55,6 +56,7 @@ $stackKeys = [ 'blog_stack_15', 'blog_stack_16', 'blog_stack_17', + 'blog_stack_18', ]; $caseStudyCards = [ @@ -167,6 +169,11 @@ $stackSourceLinks = [ ['label' => 'Apache/FPM config', 'path' => 'apps/website/httpd-php-fpm.conf'], ['label' => 'edge nginx template', 'path' => 'bootstrap/edge/templates/default.conf.tftpl'], ], + 'blog_stack_18' => [ + ['label' => 'supply-chain policy', 'path' => 'apps/supply-chain-policy/local-registry-image-policy.yaml'], + ['label' => 'Cosign build flow', 'path' => 'lab.sh'], + ['label' => 'apps variables', 'path' => 'bootstrap/apps/variables.tf'], + ], ]; function renderStackSourceLinks(string $stackKey, array $sourceLinks, string $sourceBase): void { diff --git a/apps/website/lang/en.php b/apps/website/lang/en.php index 6eabcb3..876d07d 100644 --- a/apps/website/lang/en.php +++ b/apps/website/lang/en.php @@ -149,6 +149,7 @@ return [ 'blog_stack_15' => 'OpenWrt is handled separately from the Debian golden-node template. The lab downloads the upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, and keeps it disabled unless LAB_OPENWRT_VM=true is set.', 'blog_stack_16' => 'The monitoring layer now includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, and kube-state-metrics, with placement guardrails so platform add-ons stop defaulting to the control plane.', 'blog_stack_17' => 'The portfolio website keeps Apache at the core, runs PHP through PHP-FPM with OPcache, pre-renders read-heavy pages into static HTML during the image build, and leaves PHP for write and translation endpoints.', + 'blog_stack_18' => 'The supply-chain path now uses BuildKit SBOM attestations, Cosign signatures, a cluster-published public key, and a Kyverno ImageValidatingPolicy that denies unsigned local-registry pods.', 'blog_arch_kicker' => 'Architecture map', 'blog_arch_title' => 'The homelab, end to end', 'blog_arch_intro' => 'The current delivery path starts with a push to Gitea, runs local validation, builds arm64 images, syncs the validated commit into the GitOps mirror, and lets Argo CD reconcile from the app workers. The infrastructure path stays manual through lab.sh, including the PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, and the opt-in OpenWrt firewall VM, while the OCI edge routes public traffic back through the private path.', @@ -172,6 +173,7 @@ return [ 'blog_activity_13' => 'Installed the monitoring stack and moved platform add-ons such as Argo CD, Kyverno, and prometheus-stack work toward app-worker placement instead of treating the control plane as spare capacity.', 'blog_activity_14' => 'Reworked the website image to keep Apache as the web server while moving PHP execution to PHP-FPM, enabling OPcache, and validating the runtime through Gitea Actions instead of local image builds.', 'blog_activity_15' => 'Pre-rendered the read-heavy website pages into static HTML, moved visitor ideas behind a small JSON endpoint, added cache headers, enabled HTTP/2 and connection reuse at the OCI edge, and measured reused HTTP/2 asset loads around 60ms.', + 'blog_activity_16' => 'Added image signing and SBOM enforcement: lab.sh now builds with SBOM metadata, signs php-website and demos-static with Cosign, attaches signed SPDX SBOM attestations, and Kyverno denies unsigned local-registry pods.', 'blog_todo_kicker' => 'Improvement backlog', 'blog_todo_title' => 'Todo list for the next homelab pass', 'blog_todo_intro' => 'These are improvement proposals, not chores for the sake of chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, or makes the platform easier to operate when something fails.', @@ -179,7 +181,7 @@ return [ 'blog_todo_2' => 'Keep the Debian bare GitOps mirror as the cluster source and add object-storage backups when OCI storage is ready.', 'blog_todo_3' => 'Add a real OpenTofu remote state backend with backup, locking, and a documented recovery path.', 'blog_todo_4' => 'Replace the remaining mutable latest image references with immutable tags or digest pins for demo workloads; the website image now uses a content-hash tag.', - 'blog_todo_5' => 'Generate SBOMs and sign images so the local registry can prove what it is serving.', + 'blog_todo_5' => 'Move the Cosign private key into Vault or another secret manager so the runner no longer owns long-lived signing material on disk.', 'blog_todo_6' => 'Add Renovate or Dependabot-style dependency updates for base images, Helm charts, and GitHub/Gitea Actions.', 'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, and documented exceptions for platform components.', 'blog_todo_8' => 'Turn the installed observability stack into useful operations views: a few high-signal dashboards, alerts for node health, storage pressure, certificate expiry, and failed app syncs.', diff --git a/apps/website/lang/nah.php b/apps/website/lang/nah.php index d090ce1..7af85c2 100644 --- a/apps/website/lang/nah.php +++ b/apps/website/lang/nah.php @@ -149,6 +149,7 @@ return [ 'blog_stack_15' => 'OpenWrt handled separately from Debian golden-node template. Lab downloads upstream ARM SystemReady EFI image, imports it as VM 9050 on nvme_thin_pool, ihuan keeps disabled unless LAB_OPENWRT_VM=true.', 'blog_stack_16' => 'Monitoring layer axcan includes Prometheus Stack, Grafana, Loki, Promtail, node-exporter, ihuan kube-state-metrics, ika placement guardrails para platform add-ons ma amo default to control plane.', 'blog_stack_17' => 'Portfolio website keeps Apache at core, runs PHP through PHP-FPM ika OPcache, pre-renders read-heavy pages into static HTML during image build, ihuan leaves PHP para write ihuan translation endpoints.', + 'blog_stack_18' => 'Supply-chain path axcan uses BuildKit SBOM attestations, Cosign signatures, public key ipan cluster, ihuan Kyverno ImageValidatingPolicy in quidenies unsigned local-registry pods.', 'blog_arch_kicker' => 'Architecture mapa', 'blog_arch_title' => 'Homelab, end to end', 'blog_arch_intro' => 'Current delivery path starts ika push to Gitea, runs local validation, builds arm64 images, syncs validated commit into GitOps mirror, ihuan lets Argo CD reconcile from app workers. Infrastructure path stays manual through lab.sh, including PXE/Pimox template builder, NVMe-backed worker clones, Kyverno policy placement, ihuan opt-in OpenWrt firewall VM, while OCI edge routes public traffic back through private path.', @@ -172,6 +173,7 @@ return [ 'blog_activity_13' => 'Installed monitoring stack ihuan moved platform add-ons such as Argo CD, Kyverno, ihuan prometheus-stack toward app-worker placement instead of treating control plane as spare capacity.', 'blog_activity_14' => 'Reworked website image para Apache ma mocahua web server while PHP runs through PHP-FPM, OPcache enabled, ihuan runtime validated through Gitea Actions instead of local image builds.', 'blog_activity_15' => 'Pre-rendered read-heavy website pages into static HTML, moved visitor ideas behind small JSON endpoint, added cache headers, enabled HTTP/2 ihuan connection reuse at OCI edge, ihuan measured reused HTTP/2 asset loads around 60ms.', + 'blog_activity_16' => 'Added image signing ihuan SBOM enforcement: lab.sh axcan builds ika SBOM metadata, signs php-website ihuan demos-static ika Cosign, attaches signed SPDX SBOM attestations, ihuan Kyverno quidenies unsigned local-registry pods.', 'blog_todo_kicker' => 'Improvement tlatecpanaliztli', 'blog_todo_title' => 'Todo list para next homelab pass', 'blog_todo_intro' => 'Inin cateh improvement proposals, amo chores zan pampa chores. Each item either reduces rebuild risk, tightens supply-chain hygiene, o makes platform easier to operate quema tlein fails.', @@ -179,7 +181,7 @@ return [ 'blog_todo_2' => 'Keep Debian bare GitOps mirror as cluster source ihuan add object-storage backups quema OCI storage ready.', 'blog_todo_3' => 'Add real OpenTofu remote state backend ika backup, locking, ihuan documented recovery path.', 'blog_todo_4' => 'Replace remaining mutable latest image references ika immutable tags o digest pins para demo workloads; website image axcan uses content-hash tag.', - 'blog_todo_5' => 'Generate SBOMs ihuan sign images so local registry can prove tlein serving.', + 'blog_todo_5' => 'Move Cosign private key into Vault o occe secret manager so runner ayacmo owns long-lived signing material on disk.', 'blog_todo_6' => 'Add Renovate o Dependabot-style dependency updates para base images, Helm charts, ihuan GitHub/Gitea Actions.', 'blog_todo_7' => 'Expand Kyverno baseline policy coverage: non-root, read-only roots, resource requests, allowed registries, ihuan documented exceptions para platform components.', 'blog_todo_8' => 'Turn installed observability stack into useful operations views: few high-signal dashboards, alerts para node health, storage pressure, certificate expiry, ihuan failed app syncs.', diff --git a/bootstrap/apps/variables.tf b/bootstrap/apps/variables.tf index 6a0bb9e..9dd36ff 100644 --- a/bootstrap/apps/variables.tf +++ b/bootstrap/apps/variables.tf @@ -66,5 +66,14 @@ variable "applications" { self_heal = true create_namespace = true } + supply-chain-policy = { + project = "default" + path = "apps/supply-chain-policy" + namespace = "kyverno" + target_revision = "main" + prune = true + self_heal = true + create_namespace = false + } } } diff --git a/lab.sh b/lab.sh index 1a5442a..3e524fa 100755 --- a/lab.sh +++ b/lab.sh @@ -4,6 +4,14 @@ set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BUILDX_CONFIG="/tmp/buildx-config.toml" KUBECONFIG_PATH="${KUBECONFIG_PATH:-${TF_VAR_kubeconfig_path:-/home/jv/.kube/config}}" +COSIGN_KEY_PREFIX="${COSIGN_KEY_PREFIX:-${REPO_ROOT}/.lab/cosign}" +COSIGN_KEY_PATH="${COSIGN_KEY_PATH:-${COSIGN_KEY_PREFIX}.key}" +COSIGN_PUBLIC_KEY_PATH="${COSIGN_PUBLIC_KEY_PATH:-${COSIGN_KEY_PREFIX}.pub}" +COSIGN_PASSWORD_FILE="${COSIGN_PASSWORD_FILE:-${REPO_ROOT}/.lab/cosign.password}" +COSIGN_VERSION="${COSIGN_VERSION:-2.6.3}" +COSIGN_BIN="${COSIGN_BIN:-}" +HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP="${HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP:-homelab-cosign-public-key}" +HOMELAB_SBOM_PREDICATE_TYPE="${HOMELAB_SBOM_PREDICATE_TYPE:-https://spdx.dev/Document}" trap 'rm -f "${BUILDX_CONFIG}"' EXIT @@ -182,6 +190,14 @@ adopt_apps_existing_resources() { "argoproj.io/v1alpha1" \ "Application" \ "heimdall" + adopt_tofu_kubernetes_manifest \ + "${stack}" \ + 'kubernetes_manifest.argocd_application["supply-chain-policy"]' \ + "${namespace}" \ + "applications.argoproj.io" \ + "argoproj.io/v1alpha1" \ + "Application" \ + "supply-chain-policy" } ensure_homelab_node_labels() { @@ -1613,6 +1629,218 @@ EOF docker buildx inspect --bootstrap } +ensure_cosign_available() { + local arch + local cosign_url + local target_dir + local target_path + local tmp_path + + if [[ -n "${COSIGN_BIN}" ]]; then + if [[ -x "${COSIGN_BIN}" ]]; then + return 0 + fi + echo "COSIGN_BIN points to ${COSIGN_BIN}, but it is not executable." >&2 + exit 1 + fi + + if command -v cosign >/dev/null 2>&1; then + COSIGN_BIN="$(command -v cosign)" + return 0 + fi + + case "$(uname -m)" in + x86_64 | amd64) + arch="amd64" + ;; + aarch64 | arm64) + arch="arm64" + ;; + *) + echo "Unsupported Cosign install architecture: $(uname -m)" >&2 + exit 1 + ;; + esac + + target_dir="${REPO_ROOT}/.lab/bin" + target_path="${target_dir}/cosign-v${COSIGN_VERSION}-linux-${arch}" + if [[ -x "${target_path}" ]]; then + COSIGN_BIN="${target_path}" + return 0 + fi + + if ! command -v curl >/dev/null 2>&1; then + echo "curl is required to download Cosign ${COSIGN_VERSION}." >&2 + exit 1 + fi + + mkdir -p "${target_dir}" + tmp_path="${target_path}.tmp" + cosign_url="https://github.com/sigstore/cosign/releases/download/v${COSIGN_VERSION}/cosign-linux-${arch}" + echo "Installing Cosign ${COSIGN_VERSION} into ${target_path}..." + curl -fsSL "${cosign_url}" -o "${tmp_path}" + chmod 0755 "${tmp_path}" + mv "${tmp_path}" "${target_path}" + COSIGN_BIN="${target_path}" +} + +ensure_cosign_password() { + if [[ -n "${COSIGN_PASSWORD:-}" ]]; then + return 0 + fi + + if [[ -r "${COSIGN_PASSWORD_FILE}" ]]; then + COSIGN_PASSWORD="$(<"${COSIGN_PASSWORD_FILE}")" + export COSIGN_PASSWORD + return 0 + fi + + mkdir -p "$(dirname "${COSIGN_PASSWORD_FILE}")" + if command -v openssl >/dev/null 2>&1; then + openssl rand -hex 32 > "${COSIGN_PASSWORD_FILE}" + elif command -v python3 >/dev/null 2>&1; then + python3 - <<'PY' > "${COSIGN_PASSWORD_FILE}" +import secrets + +print(secrets.token_hex(32)) +PY + else + echo "openssl or python3 is required to generate a non-interactive Cosign key password." >&2 + exit 1 + fi + chmod 0600 "${COSIGN_PASSWORD_FILE}" + COSIGN_PASSWORD="$(<"${COSIGN_PASSWORD_FILE}")" + export COSIGN_PASSWORD +} + +run_cosign() { + ensure_cosign_available + ensure_cosign_password + + COSIGN_PASSWORD="${COSIGN_PASSWORD}" "${COSIGN_BIN}" "$@" +} + +ensure_cosign_keypair() { + ensure_cosign_available + ensure_cosign_password + + if [[ -f "${COSIGN_KEY_PATH}" && -f "${COSIGN_PUBLIC_KEY_PATH}" ]]; then + return 0 + fi + + if [[ -f "${COSIGN_KEY_PATH}" || -f "${COSIGN_PUBLIC_KEY_PATH}" ]]; then + echo "Found only one Cosign key file. Expected both ${COSIGN_KEY_PATH} and ${COSIGN_PUBLIC_KEY_PATH}." >&2 + exit 1 + fi + + mkdir -p "$(dirname "${COSIGN_KEY_PATH}")" + echo "Generating homelab Cosign key pair under ${REPO_ROOT}/.lab..." + COSIGN_PASSWORD="${COSIGN_PASSWORD}" "${COSIGN_BIN}" generate-key-pair --output-key-prefix "${COSIGN_KEY_PREFIX}" + chmod 0600 "${COSIGN_KEY_PATH}" + chmod 0644 "${COSIGN_PUBLIC_KEY_PATH}" +} + +publish_cosign_public_key_config_map() { + kubectl --kubeconfig "${KUBECONFIG}" create namespace kyverno --dry-run=client -o yaml | + kubectl --kubeconfig "${KUBECONFIG}" apply -f - + + kubectl --kubeconfig "${KUBECONFIG}" -n kyverno create configmap "${HOMELAB_COSIGN_PUBLIC_KEY_CONFIGMAP}" \ + --from-file=cosign.pub="${COSIGN_PUBLIC_KEY_PATH}" \ + --dry-run=client -o yaml | + kubectl --kubeconfig "${KUBECONFIG}" apply -f - +} + +cosign_registry_flags() { + local registry_endpoint="$1" + local image_ref="$2" + + if [[ "${image_ref}" == "${registry_endpoint}/"* ]]; then + printf '%s\n' "--allow-http-registry" + fi +} + +image_supply_chain_metadata_exists() { + local image_ref="$1" + local registry_endpoint="$2" + local -a registry_flags=() + + mapfile -t registry_flags < <(cosign_registry_flags "${registry_endpoint}" "${image_ref}") + + run_cosign verify \ + --key "${COSIGN_PUBLIC_KEY_PATH}" \ + --insecure-ignore-tlog=true \ + "${registry_flags[@]}" \ + "${image_ref}" >/dev/null 2>&1 && + run_cosign verify-attestation \ + --key "${COSIGN_PUBLIC_KEY_PATH}" \ + --type "${HOMELAB_SBOM_PREDICATE_TYPE}" \ + --insecure-ignore-tlog=true \ + "${registry_flags[@]}" \ + "${image_ref}" >/dev/null 2>&1 +} + +write_image_sbom_predicate() { + local image_ref="$1" + local output_file="$2" + + docker buildx imagetools inspect "${image_ref}" --format '{{ json .SBOM.SPDX }}' > "${output_file}" + python3 - "${output_file}" <<'PY' +import json +import sys + +path = sys.argv[1] +with open(path, encoding="utf-8") as handle: + document = json.load(handle) + +if not isinstance(document, dict): + raise SystemExit("image SBOM predicate is not a JSON object") +if document.get("SPDXID") != "SPDXRef-DOCUMENT": + raise SystemExit("image SBOM predicate is not an SPDX document") +if not document.get("packages"): + raise SystemExit("image SBOM predicate does not include packages") +PY +} + +publish_image_supply_chain_metadata() { + local image_ref="$1" + local registry_endpoint="$2" + local source_hash="$3" + local sbom_file + local -a registry_flags=() + + if image_supply_chain_metadata_exists "${image_ref}" "${registry_endpoint}"; then + echo "Image ${image_ref} already has a valid Cosign signature and signed SPDX SBOM." + return 0 + fi + + mapfile -t registry_flags < <(cosign_registry_flags "${registry_endpoint}" "${image_ref}") + sbom_file="$(mktemp)" + write_image_sbom_predicate "${image_ref}" "${sbom_file}" + + echo "Signing image ${image_ref} and attaching signed SPDX SBOM..." + run_cosign sign \ + --yes \ + --key "${COSIGN_KEY_PATH}" \ + --tlog-upload=false \ + -a "homelab.dev/source-hash=${source_hash}" \ + "${registry_flags[@]}" \ + "${image_ref}" + run_cosign attest \ + --yes \ + --key "${COSIGN_KEY_PATH}" \ + --predicate "${sbom_file}" \ + --type "${HOMELAB_SBOM_PREDICATE_TYPE}" \ + --tlog-upload=false \ + "${registry_flags[@]}" \ + "${image_ref}" + rm -f "${sbom_file}" + + if ! image_supply_chain_metadata_exists "${image_ref}" "${registry_endpoint}"; then + echo "Cosign verification failed after signing ${image_ref}." >&2 + exit 1 + fi +} + dump_argocd_debug() { local app="$1" @@ -1669,6 +1897,24 @@ wait_for_namespaced_resource() { done } +wait_for_cluster_resource() { + local kind="$1" + local name="$2" + local app="$3" + local timeout_seconds="$4" + local elapsed=0 + + until kubectl --kubeconfig "${KUBECONFIG}" get "${kind}/${name}" >/dev/null 2>&1; do + if ((elapsed >= timeout_seconds)); then + echo "Timed out waiting for ${kind}/${name} from Argo CD app ${app}" >&2 + dump_argocd_debug "${app}" + exit 1 + fi + sleep 5 + elapsed=$((elapsed + 5)) + done +} + wait_for_deployment_ready() { local namespace="$1" local deployment="$2" @@ -2595,6 +2841,9 @@ apps() { export TF_VAR_kubeconfig_path="${TF_VAR_kubeconfig_path:-${KUBECONFIG_PATH}}" export KUBECONFIG="${TF_VAR_kubeconfig_path}" + ensure_cosign_keypair + publish_cosign_public_key_config_map + if [[ "${TF_VAR_registry_endpoint}" != "${registry_endpoint}" ]]; then echo "TF_VAR_registry_endpoint changed after registry endpoint resolution (${registry_endpoint})" >&2 exit 1 @@ -2621,14 +2870,15 @@ apps() { run_tofu_stack "bootstrap/apps" refresh_argocd_application container-registry - refresh_argocd_application demos-static - refresh_argocd_application website-production + refresh_argocd_application supply-chain-policy + wait_for_cluster_resource imagevalidatingpolicy.policies.kyverno.io homelab-local-registry-supply-chain supply-chain-policy 300 wait_for_namespace container-registry container-registry 300 wait_for_namespaced_resource container-registry deployment local-registry container-registry 300 wait_for_deployment_ready container-registry local-registry container-registry 300 - if website_image_is_current "${website_image_state_file}" "${website_source_hash}" "${website_platforms}" "${website_image_ref}" "${registry_endpoint}"; then + if website_image_is_current "${website_image_state_file}" "${website_source_hash}" "${website_platforms}" "${website_image_ref}" "${registry_endpoint}" && + image_supply_chain_metadata_exists "${website_image_ref}" "${registry_endpoint}"; then echo "Website image ${website_image_ref} is already current (${website_source_hash}); skipping build." else echo "Building website image ${website_image_ref} for ${website_platforms} (${website_source_hash})..." @@ -2642,7 +2892,7 @@ apps() { --network host \ --platform "${website_platforms}" \ --provenance=false \ - --sbom=false \ + --sbom=true \ --label "dev.homelab.website.source-hash=${website_source_hash}" \ -t "${website_image_ref}" \ -f "${REPO_ROOT}/apps/website/Dockerfile" \ @@ -2650,8 +2900,10 @@ apps() { --push website_image_built=true fi + publish_image_supply_chain_metadata "${website_image_ref}" "${registry_endpoint}" "${website_source_hash}" - if demos_image_is_current "${demos_image_state_file}" "${demos_source_hash}" "${demos_platforms}" "${demos_image_ref}" "${registry_endpoint}"; then + if demos_image_is_current "${demos_image_state_file}" "${demos_source_hash}" "${demos_platforms}" "${demos_image_ref}" "${registry_endpoint}" && + image_supply_chain_metadata_exists "${demos_image_ref}" "${registry_endpoint}"; then echo "Demos image ${demos_image_ref} is already current (${demos_source_hash}); skipping build." else echo "Building demos image ${demos_image_ref} for ${demos_platforms} (${demos_source_hash})..." @@ -2665,7 +2917,7 @@ apps() { --network host \ --platform "${demos_platforms}" \ --provenance=false \ - --sbom=false \ + --sbom=true \ --label "dev.homelab.demos.source-hash=${demos_source_hash}" \ -t "${demos_image_ref}" \ -f "${REPO_ROOT}/apps/demos-static/Dockerfile" \ @@ -2673,6 +2925,7 @@ apps() { --push demos_image_built=true fi + publish_image_supply_chain_metadata "${demos_image_ref}" "${registry_endpoint}" "${demos_source_hash}" refresh_argocd_application website-production wait_for_namespace website-production website-production 300