SLSA for Models on GCP

Prior to this change, SLSA for Models targeted GitHub Actions only. In this change, we demonstrate SLSA for Models on GCP using Tekton, Sigstore, Google Kubernetes Engine and Artifact Registry.
sigstore · Jan 10, 2024 · 8c134fd · 8c134fd
1 parent 1f5c5fb
commit 8c134fd
Show file tree

Hide file tree

Showing 9 changed files with 804 additions and 61 deletions.
diff --git a/README.md b/README.md
@@ -68,16 +68,12 @@ See [model_signing/README.md](model_signing/README.md) for more information.
 
 ### SLSA for ML
 
-This project shows how we can use the [SLSA L3 GitHub generator][slsa-generator]
-to generate SLSA provenance for ML models. The SLSA generator was originally
-developed for traditional software to protect against tampering with builds,
-such as in the [Solarwinds attack][solarwinds], and this project is a proof of
-concept that the same supply chain protections can be applied to ML.
+This project shows how we can generate [SLSA][slsa] provenance for ML models.
 
-While most of the ML models are currently too expensive to train using the SLSA
-generator on GitHub Actions, future work will cover the training of ML models
-that require access to accelerators (i.e., GPUs, TPUs) or that require multiple
-hours for training.
+SLSA was originally developed for traditional software to protect against
+tampering with builds, such as in the [Solarwinds attack][solarwinds], and
+this project is a proof of concept that the same supply chain protections
+can be applied to ML.
 
 See [slsa_for_models/README.md](slsa_for_models/README.md) for more information.
 
@@ -91,6 +87,7 @@ used in any production environment.
 
 Please see the [Contributor Guide](CONTRIBUTING.md) for more information.
 
+[slsa]: https://slsa.dev/
 [saif]: https://blog.google/technology/safety-security/introducing-googles-secure-ai-framework/
 [openssf]: https://openssf.org/
 [slsa-generator]: https://github.com/slsa-framework/slsa-github-generator

diff --git a/slsa_for_models/README.md b/slsa_for_models/README.md
@@ -1,34 +1,29 @@
 # SLSA for Models
 
-This project shows how we can use the [SLSA L3 GitHub generator][slsa-generator]
-to generate SLSA provenance for ML models. The SLSA generator was originally
-developed for traditional software to protect against tampering with builds,
-such as in the [Solarwinds attack][solarwinds], and this project is a proof of
-concept that the _same supply chain protections can be applied to ML_.
+This project shows how we can generate [SLSA][slsa] provenance for ML models
+on [GitHub Actions][gha] and [Google Cloud Platform][gcp].
 
-Future work will involve covering training ML models that require access to
-accelerators (i.e., GPUs, TPUs) or that require multiple hours for training. As
-an example, we could use [Tekton Chains][tekton-chains] to support [training ML
-models using Kubeflow][tekton-kubeflow].
+SLSA was originally developed for traditional software to protect against
+tampering with builds, such as in the [Solarwinds attack][solarwinds], and
+this project is a proof of concept that the _same supply chain protections
+can be applied to ML_.
 
-When users download a given version of a model they can also check its
-provenance by using [the SLSA verifier][slsa-verifier] repository. This can be
-integrated in the model hub and/or model serving platforms: for example the
-model serving pipeline could validate provenance for all new models before
-serving them. However, the verification can also be done manually, on demand, as
-we will do by the end of this README.
+When users download a given version of a model they can also check its provenance.
+This can be integrated in the model hub and/or model serving platforms: for example
+the model serving pipeline could validate provenance for all new models before
+serving them. However, the verification can also be done manually, on demand.
 
 As an additional benefit, having provenance for a model allows users to react
 to vulnerabilities in a training framework: they can quickly determine if a
 model needs to be retrained because it was created using a vulnerable version.
 
-## Usage
+See the guides for [GitHub Actions][gha] and [Google Cloud Platform][gcp] for details.
+
+## Models
 
 We support both TensorFlow and PyTorch models. The example repo trains a model
 on [CIFAR10][cifar10] dataset, saves it in one of the supported formats, and
-generates provenance for the output. All of this happens during a [GitHub Actions
-workflow][workflow] which takes as input the format to save the model into. The
-supported formats are:
+generates provenance for the output. The supported formats are:
 
 | Workflow Argument            | Training Framework | Model format                    |
 |------------------------------|--------------------|---------------------------------|
@@ -39,40 +34,24 @@ supported formats are:
 | `pytorch_full_model.pth`     | PyTorch            | PyTorch complete model format   |
 | `pytorch_jitted_model.pt`    | PyTorch            | PyTorch TorchScript format      |
 
-To test, fork this repository, then head over to the Actions tab and select the
-"SLSA for ML models example" workflow. Since the workflow has a
-`workflow_dispatch` trigger, it can be invoked on demand: click the `Run
-workflow` button, then select the value for the "Name of the model" argument.
-
-![Triggering a SLSA workflow](images/slsa_trigger.png)
-
-After the workflow finishes execution, there will be two archives in the
-"Artifacts" section: one is the model that was trained and the other one is the
-SLSA provenance attached to the model.
+While most of the ML models are currently too expensive to train, future work will
+cover the training of ML models that require access to accelerators (i.e., GPUs, TPUs)
+or that require multiple hours for training.
 
-![Results of running a SLSA workflow](images/slsa_results.png)
+## Future Work
 
-To verify the provenance, download both archives, unzip each and then run
-`slsa-verifier`, making sure to replace the `--source-uri` argument with the
-_path to your fork_. For example, for a PyTorch model, which has been [built on
-this repository](https://github.com/google/model-transparency/actions/runs/6646816974):
-
-```console
-[...]$ slsa-verifier verify-artifact \
-       --provenance-path pytorch_model.pth.intoto.jsonl \
-       --source-uri github.com/google/model-transparency \
-       pytorch_model.pth
-Verified signature against tlog entry index 45419124 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77a98dd03d23a78657e7f1efd3d9bea6988abbf23a72290a4ec7dc35c9edeab7ee1
-Verified build using builder "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.9.0" at commit ac26cbf66849cfec6f29747f4525180595c7eef0
-Verifying artifact pytorch_model.pth: PASSED
-
-PASSED: Verified SLSA provenance
-```
+### Accelerators
+Future work will involve covering training ML models that require access to
+accelerators (i.e., GPUs, TPUs) or that require multiple hours for training. As
+an example, we could use [Tekton Chains][tekton-chains] to support [training ML
+models using Kubeflow][tekton-kubeflow].
 
-The verification of provenance can be done just before the model gets loaded in the
-serving pipeline.
+### Platforms
+While this example has targeted GitHub Actions and Tekton in GCP, we aim to bring
+support for other platforms (e.g., GCB and GitLab) and model training environments.
 
-**Note**: TensorFlow also supports saving models in `SavedModel` format. This is
+### Directory Format
+TensorFlow also supports saving models in `SavedModel` format. This is
 a directory-based serialization format and currently we don't fully support
 this. We can generate SLSA provenance for all the files in the directory but
 there are caveats regarding verification. Furthermore, because there is a
@@ -81,9 +60,6 @@ during model signing, we have decided to add support for these model formats at
 a future time, after standardizing a way to generate and verify provenance in
 SLSA (in general, not just for ML).
 
-While this example has targeted GitHub Actions, we aim to bring support
-for other platforms (e.g., GCB and GitLab) and model training environments.
-
 [cifar10]: https://www.cs.toronto.edu/~kriz/cifar.html
 [slsa-generator]: https://github.com/slsa-framework/slsa-github-generator
 [slsa-verifier]: https://github.com/slsa-framework/slsa-verifier/
@@ -92,3 +68,5 @@ for other platforms (e.g., GCB and GitLab) and model training environments.
 [tekton-chains]: https://github.com/tektoncd/chains
 [tekton-kubeflow]: https://www.kubeflow.org/docs/components/pipelines/v1/sdk/pipelines-with-tekton/
 [workflow]: https://github.com/google/model-transparency/blob/main/.github/workflows/slsa_for_ml.yml
+[gha]: github_actions
+[gcp]: gcp
diff --git a/slsa_for_models/gcp/README.md b/slsa_for_models/gcp/README.md
@@ -0,0 +1,247 @@
+# SLSA for Models on Google Cloud Platform
+
+This project uses [Tekton][tekton] to generate SLSA provenance for ML models on Google Cloud Platform (GCP). It uses 
+[Google Kubernetes Engine][gke] (GKE), [Artifact Registry][ar], [Tekton] and [Sigstore].
+
+## Guide
+
+1. To get started, you'll need to have a [GCP Project][gcp]. You will also need to have these CLI tools installed:
+   - [`gcloud`][gcloud]
+   - [`kubectl`][kubectl]
+   - [`tkn`][tkn]
+   - [`cosign`][cosign]
+
+2. Enable the needed services:
+
+   ```shell
+   gcloud services enable \
+     container.googleapis.com \
+     artifactregistry.googleapis.com
+   ```
+
+3. Create a GKE cluster:
+
+    1. Set the `PROJECT_ID` environment variable from your GCP project:
+
+       ```shell
+       export PROJECT_ID=<PROJECT_ID>
+       ```
+
+   2. Set the `CLUSTER_NAME` environment variable to a cluster name of your choice:
+
+      ```shell
+      export CLUSTER_NAME=<CLUSTER_NAME>
+      ```
+
+   3. Create a cluster:
+
+       ```shell
+       gcloud container clusters create $CLUSTER_NAME \
+       --enable-autoscaling \
+       --min-nodes=1 \
+       --max-nodes=3 \
+       --scopes=cloud-platform \
+       --no-issue-client-certificate \
+       --project=$PROJECT_ID \
+       --region=us-central1 \
+       --machine-type=e2-standard-4 \
+       --num-nodes=1 \
+       --cluster-version=latest
+       ```
+
+4. Install Tekton:
+
+   1. Install Tekton Pipelines:
+
+       ```shell
+       kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
+       ``` 
+
+   2. Install Tekton Chains:
+
+       ```shell
+       kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml
+       ```
+
+5. Verify your Tekton installation was successful:
+
+   1. Check that Tekton Pipelines Pods are running in Kubernetes:
+
+       ```shell
+       kubectl get pods -n tekton-pipelines
+       ```
+
+   2. Check that Tekton Chains Pods are running in Kubernetes:
+
+      ```shell
+      kubectl get pods -n tekton-chains 
+      ``` 
+
+6. Configure Tekton:
+
+   1. Configure Tekton Pipelines to enable enumerations and alpha features:
+
+       ```shell
+       kubectl patch cm feature-flags -n tekton-pipelines -p '{"data":{
+       "enable-param-enum":"true",
+       "enable-api-fields":"alpha" 
+       }}'  
+       ```
+
+   2. Then restart the Tekton Pipelines controller to ensure it picks up the changes:
+
+      ```shell
+      kubectl delete pods -n tekton-pipelines -l app=tekton-pipelines-controller
+      ``` 
+
+   3. Configure Tekton Chains to enable transparency log, set SLSA format and configure storage:
+
+      ```shell
+      kubectl patch configmap chains-config -n tekton-chains -p='{"data":{
+      "transparency.enabled": "true",
+      "artifacts.taskrun.format":"slsa/v2alpha2", 
+      "artifacts.taskrun.storage": "tekton",
+      "artifacts.pipelinerun.format":"slsa/v2alpha2", 
+      "artifacts.pipelinerun.storage": "tekton"
+      }}'
+      ```
+   4. Then restart the Tekton Chains controller to ensure it picks up the changes:
+
+      ```shell
+      kubectl delete pods -n tekton-chains -l app=tekton-chains-controller 
+      ```    
+
+7. Generate an encrypted x509 keypair and save it as a Kubernetes secret:
+
+   ```shell
+   cosign generate-key-pair k8s://tekton-chains/signing-secrets
+   ```
+
+8. Apply the `Tasks`:
+
+   ```shell
+   kubectl apply -f slsa_for_models/gcp/tasks
+   ```
+
+9. Apply the `Pipeline`:
+
+   ```shell
+   kubectl apply -f slsa_for_models/gcp/pipeline.yml
+   ```
+
+10. Create a generic repository in Artifact Registry:
+
+    1. Set the `REPOSITORY_NAME` environment variable to a name of your choice:
+
+       ```shell
+       export REPOSITORY_NAME=ml-artifacts
+       ```
+
+    2. Set the `LOCATION` environment variable to a [location] of your choice:
+
+       ```shell
+       export LOCATION=us
+       ```
+
+    3. Create a generic repository:
+        ```shell
+        gcloud artifacts repositories create $REPOSITORY_NAME \
+         --location=$LOCATION \
+         --repository-format=generic
+        ```   
+
+    4. If you set a different repository name and location from the example above, make sure to modify the `Parameter`
+    named 'model-storage' in the `PipelineRun` with your own values. 
+
+11. Execute the `PipelineRun`:
+
+    ```shell
+    kubectl create -f slsa_for_models/gcp/pipelinerun.yml
+    ```
+
+12. Observe the `PipelineRun` execution:
+
+    ```shell
+    export PIPELINERUN_NAME=$(tkn pr describe --last --output jsonpath='{.metadata.name}')
+    tkn pipelinerun logs $PIPELINERUN_NAME --follow
+    ``` 
+
+13. When the `PipelineRun` succeeds, view its status:
+
+    ```shell
+    kubectl get pipelinerun $PIPELINERUN_NAME --output yaml
+    ```
+
+14. View the transparency log entry in the public [Rekor][rekor] instance:
+
+   ```shell
+   export TLOG_ENTRY=$(tkn pr describe $PIPELINERUN_NAME --output jsonpath="{.metadata.annotations.chains\.tekton\.dev/transparency}")
+   open $TLOG_ENTRY
+   ```
+
+15. Retrieve the attestation from the `PipelineRun` which is stored as a base64-encoded annotation:
+
+   ```shell
+   export PIPELINERUN_UID=$(tkn pr describe $PIPELINERUN_NAME --output  jsonpath='{.metadata.uid}')
+   tkn pr describe $PIPELINERUN_NAME --output jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-pipelinerun-$PIPELINERUN_UID}" | base64 -d > pytorch_model.pth.build-slsa 
+   ```
+
+16. View the attestation:
+
+   ```shell
+   cat pytorch_model.pth.build-slsa | tr -d '\n' | pbcopy
+   pbpaste | jq '.payload | @base64d | fromjson'
+   ```
+17. Download the model:
+
+   ```shell
+   export MODEL_VERSION=$(tkn pr describe slsa-for-models-pr --output jsonpath='{.status.results[1].value.digest}' | cut -d ':' -f 2)
+   gcloud artifacts generic download \
+   --package=pytorch-model \
+   --repository=$REPOSITORY_NAME \
+   --destination=. \
+   --version="${MODEL_VERSION:7}"
+   ```
+
+18. Verify the attestation:
+
+   ```shell
+   cosign verify-blob-attestation \
+    --key k8s://tekton-chains/signing-secrets \
+    --signature pytorch_model.pth.build-slsa \
+    --type slsaprovenance1 \
+    pytorch_model.pth
+   ```
+
+## Future Work
+
+### Automate Provenance Verification
+
+Demonstrate how to verify the provenance of the model before deploying and serving the model.
+
+### Automated Testing
+
+Trigger execution of the `PipelineRun` whenever changes are made in the codebase.
+
+### Kubeflow on Tekton
+
+Provide a Kubeflow Pipeline that can be compiled into the above Tekton Pipeline using [Kubeflow on Tekton][tekton-kubeflow].
+
+### Accelerators
+
+Demonstrate training ML models that require multiple hours for training and require access to accelerators (i.e., GPUs, 
+TPUs).
+
+[gcp]: https://cloud.google.com/docs/get-started
+[gcloud]: https://cloud.google.com/sdk/docs/install
+[kubectl]: https://kubernetes.io/docs/tasks/tools/
+[tkn]: https://tekton.dev/docs/cli/
+[cosign]: https://docs.sigstore.dev/system_config/installation/
+[tekton-kubeflow]: https://www.kubeflow.org/docs/components/pipelines/v1/sdk/pipelines-with-tekton/
+[tekton-chains]: https://tekton.dev/docs/chains/
+[tekton]: https://tekton.dev/docs/
+[rekor]: https://rekor.sigstore.dev
+[location]: https://cloud.google.com/artifact-registry/docs/repositories/repo-locations
+[gke]: https://cloud.google.com/kubernetes-engine?hl=en
+[ar]: https://cloud.google.com/artifact-registry
+[sigstore]: https://docs.sigstore.dev