Skip to content

Patch/template docs #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

Patch/template docs #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
9b2cd71
adding template for custom image handling
Aug 1, 2022
dde42e6
adding byoc setup
Aug 2, 2022
5c06acd
fixed script
Aug 2, 2022
ddaf68c
adding more seed code and upating the docker build script
Aug 2, 2022
b552235
fixes to typos and iam
georschi Aug 2, 2022
7cb9ef9
fix some errors in the code and added a read me for advanced topics
Aug 3, 2022
46fa879
byoc changes
georschi Aug 3, 2022
8fb9631
renamed folder name to mlops_sm_project_template
Aug 4, 2022
e78ee0b
modified description
Aug 4, 2022
befac29
fixes to BYOC seed code
georschi Aug 5, 2022
cd95d70
Merge branch 'feature/custom-image-template' of https://github.com/Fa…
georschi Aug 5, 2022
a0399a4
reformatting and adding more docs
Aug 5, 2022
fee1150
more docs
Aug 5, 2022
4bb2651
more docs
Aug 5, 2022
0dd2077
first commit
Aug 5, 2022
d1e8b14
minor bug fixes
Aug 5, 2022
7a99206
adding more docs
Aug 25, 2022
f6804df
add draft docs for BYOC
Aug 25, 2022
d13482f
still missing some docs
Aug 25, 2022
71269e3
Merge branch 'main' of github.com:aws-samples/sagemaker-custom-projec…
Aug 25, 2022
2ac1a0b
rename path for templates docs and add docs for creating a new template
Sep 1, 2022
903fe36
add more documentation for the dynamic account project template
Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 5 additions & 120 deletions mlops-multi-account-cdk/mlops-sm-project-template/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,6 @@ This repository contains the resources that are required to deploy the MLOps Fou

- [MLOps Foundation SageMaker Project Template](#mlops-foundation-sagemaker-project-template)
- [Solution Architecture](#solution-architecture)
- [Service Catalog Stack](#service-catalog-stack)
- [SageMaker Project Stack](#sagemaker-project-stack)
- [Shared Resources](#shared-resources)
- [Build App CI/CD Construct](#build-app-cicd-construct)
- [Deploy App CI/CD Construct](#deploy-app-cicd-construct)
- [CodeCommit Stack](#codecommit-stack)
- [Pipeline Stack](#pipeline-stack)
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Repository Structure](#repository-structure)
Expand All @@ -25,120 +18,12 @@ This repository contains the resources that are required to deploy the MLOps Fou

![mlops project architecture](diagrams/MLOPs%20Foundation%20Architecture-mlops%20project%20cicd%20architecture.jpg)

For a detailed explaination of the solution architecture go to [Solution Architecture](docs/SOLUTION_ARCHITECTURE.md)

### Service Catalog Stack

*This stack is only deployed in the DEV account*

In this stack we define the Service Catalog Portfolio and Product that we want to use for SageMaker Project Template.

The stack expects 4 cloudformation parameters:
- **Execution Role Arn:** This parameter is of type `AWS::SSM::Parameter::Value<String>` so you should provide an `SSM Parameter Name` as an input; the default value is `"/mlops/role/lead"` which is deployed as part of **mlops infrastructure repository**. Ensure you have this parameter defined in your account before you deploy the solution. This is the role that would be given visibility to the Service Catalog Portfolio and Products. Products with tag `sagemaker:studio-visibility = true` will be visible in SageMaker Studio Domain.
- **Portfolio Name:** This parameter is of type `String` and has a default value of `SageMaker Organization Templates`. It will be used for Service Catalog Portfolio resource.
- **Portfolio Owner:** This parameter is of type `String` and has a default value of `administrator`. It will be used for Service Catalog Portfolio and Product resources.
- **Product Version:** This parameter is of type `String` and has a default value of `1.0`. It will be used for Service Catalog Product resource.

The following resources are created:
- **Product Launch Role** is created which has permissions to create all the resources defined in SageMaker Project Stack and access to resources used in that stack i.e. SSM Parameters and access to CDK Assets S3 bucket (deployed as part of CDKToolkit stack).
- **A Service Catalog Product**, CloudFormation Product, for SageMaker Project Template as defined in [SageMaker Project Stack](#sagemaker-project-stack). The Product Launch Role is linked to this product.
- **A Service Catalog Portfolio**, the product would be associated to this portfolio and has visibility to sagemaker studio and linked to the Execution Role provided in the cloudformation parameters.
- **2 assets**, a zip for the **seed code** for each of the **build app** and the **deploy app**, stored in CDK Assets bucket, this bucket is created as part of **CDKToolkit stack**. A SSM Parameter is created for each asset's s3 key and also for the s3 bucket name. These parameters are used in [SageMaker Project Stack](#sagemaker-project-stack). The assets structure is documented in [MLOps Foundation Strategy](link-quip)
- **SSM Parameters** containig configuration about the target deployment accounts: **DEV**, **PREPROD** and **PROD** i.e. account id, region. These Parameters are used by the deploy app.

Once this stack is deployed in the account, the template will be usable by Amazon SageMaker Studio Domain user with the role: **Execution Role** that was provided to this stack.


### SageMaker Project Stack
*This stack is stored as a service catalog product in the DEV Account and is visible in SageMaker Studio Domain*

![project architecture](diagrams/MLOPs%20Foundation%20Architecture-sagemaker%20project%20architecture.jpg)

The ML solutions' strategy defined in this stack uses two repositories setup: **(a)** building/training repository for training and batch inference ML pipeline development, and **(b)** deployment repository to promote the batch inference pipeline models or instantiate the real time endpoints. The second repository will incorporate also the testing methods including, integration test, stress test, or custom ML tests that the data scientists want to perform to ensure the robustness of the models.

The CI/CD pipelines follow a single branch strategy with deployments to the accounts driven by commits to the `main` branch. The pipeline contains a stage for each account deployment.

The stack expects 2 cloudformation parameters:
- **Project Name:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. It will be visible in SageMaker Studio Domain and be used to then tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectName`.
- **Project ID:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. When creating a project in SageMaker Studio Domain and it will be automatically generated and then used to tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectId`.

Other parameters can be included and those will be visible in SageMaker Studio Domain during Project Creation.

All resources deployed as part of this stack are tagged with those 2 parameters.

This stack can be broken into 3 parts:
- [Shared resources with Cross Account Permissions](#shared-resources)
- [Build application resources and CI/CD pipline](#build-app-cicd-construct)
- [Deploy application resources and CI/CD pipeline](#deploy-app-cicd-construct)

#### Shared Resources
The following resources are deployed as part of a SageMaker Project and are used by both build and deploy applications:
- **Artifact Bucket** and its **KMS key**, this bucket will be used to store SageMaker Pipeline's steps outputs and also the trained model for this project.
- **Model Package Group**, this group is created at this stack mainly to setup its policy and enable cross account access to the other deployment accounts.
- **Pipeline Bucket** and its **KMS key**, this bucket will be used by both CICD pipelines to store the pipelines artifacts.

These resources are deployed to the DEV account with cross account access enabled for PREPROD and PROD; except for the pipeline bucket and its kms key, those are constraint to the DEV account.

#### Build App CI/CD Construct
*The CICD pipeline in this construct only deploys to DEV*
![build app](diagrams/building.png)

This construct contains resources that create a CI/CD pipeline to orchestrate a model training using SageMaker Pipelines starting from doing preprocessing jobs over the data and end by registering the trained model in a model package group in SageMaker Model Registry. After the pipeline finishes running successfuly, the model will have `Pending Manual Approval` status in SageMaker Model Registry.

In `seed_code/build_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based and is expected to run inside a CodeBuild project. There is a `buildspec.yml` that describes the command that will be run. For more details about this base code, refer to `seed_code/build_app/README.md`.

The following resources are created in this construct:
- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/build_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/build`. The stack expects these parameters to exist in the account.
- **CodeBuild Project**, this codebuild project is used to create/update/run the SageMaker Pipeline defined in the **build app repository** in `ml_pipelines/training/pipeline.py`. The project's **buildspec** uses the `buildspec.yml` defined in the repository as well.
- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. It has one additional stage, **Build** stage, which runs the **CodeBuild Project** defined above.

**NOTE** The solution defined above only support a CICD pipeline that is linked to the **main** branch update events. If you want to have a **multi-branch approach** i.e. a CI/CD pipeline linked to the **develop** then you will need to duplicate the exisiting pipeline and ensure resources for each branch are **isolated** to not have impact on each other. You will also need to recosider the prefix strategy used in **Artifact Bucket**, created in [shared resources section](#shared-resources).



#### Deploy App CI/CD Construct
*The CICD pipeline in this construct only deploy to DEV, PREPROD and PROD*
![build app](diagrams/deployment.png)

This construct contains resources that create a CI/CD pipeline to automate the deployment of an Amazon SageMaker Endpoint for real time inference across multiple accounts.

In `seed_code/deploy_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based **AWS CDK** application. For more details about this base code, refer to `seed_code/deploy_app/README.md`.


The following resources are created in this construct:
- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/deploy_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/deploy`. The stack expects these parameters to exist in the account.
- **EventBridge Rule**, this rule is linked to the model package group used for this SageMaker Project. It will be monitoring **Approved** and **Rejected** events and will trigger the deployment CI/CD pipeline on them.
- **CodeBuild Project for CDK Synth**, this codebuild project is used to synthesize the AWS CDK application defined in the **deploy app repository**. 3 Cloudformation Templates are expected to be generated, each template references a targeted deployment account/environment: DEV, PREPROD and PROD.
- **CodeBuild Project for CFN Nag**, this codebuild project is used to run `cfn-nag` over the cloudformation templates to ensure that they satisfy best Security practices.
- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. The stages for this pipeline are defined as such:
- A **Build** stage which runs the **CodeBuild Project for CDK Synth**
- A **Security Evaluation** stage which runs the **CodeBuild Project for CFN Nag**
- There is a stage for each account: **DEV**, **PREPROD** and **PROD** each using Cloudformation Action to deploy the templates generated in the **Build** stage. There is a manual approval action before **PREPROD** and **PROD** deployments.

There are 2 way to trigger the deployment CI/CD Pipeline:
- **Model Events** - These are events which get triggered through a status change to the model package group in SageMaker Model Registry.
- **Code Events** - The pipeline is triggered on git update events over a specific branch, in this solution it is linked to the **main** branch.

**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file.

### CodeCommit Stack
*This stack is only needed if you want to handle deployments of this folder of the repository to be managed through a CICD pipeline.*

This stack handles setting up an AWS CodeCommit repository for this folder of the repository. This repository will be used as the source for the CI/CD pipeline defined in [Pipeline Stack](#pipeline-stack). The repository will be named based on the value defined in `mlops_sm_project_template/config/constants.py` with this variable `CODE_COMMIT_REPO_NAME`. The repository will be intialised with a default branch as defined in the `constants.py` file under `PIPELINE_BRANCH` variable.

### Pipeline Stack

*This stack is only needed if you want to handle deployments of this folder of repository to be managed through a CICD pipeline. The pipeline is configured to deploy to 1 account: DEV and will deploy the service catalog stack to the target account*

The CICD pipeline in this repository is setup to monitor an AWS CodeCommit repository as defined in [CodeCommit Stack](#codecommit-stack).

If you are using other sources like github or bitbucket for your repository, you will need to modify the connection to the appropriate repository as defined in `mlops_sm_project_template/pipeline_stack.py`. This can be done using AWS CodeStar but must be setup on the account.

Make sure the pipelines also point to your targeted branch; by default the pipeline is linked to `main` branch events, this is defined in the `constants.py` file under `PIPELINE_BRANCH` variable.

`constants.py` also contains information about the target accounts you want to use for this repository CI/CD pipeline and the target deployment accounts: **DEV**, **PREPROD** and **PROD**, this information will also be deployed in SSM Parameter in the DEV account for the Deploy App CI/CD pipeline.

The pipeline will deploy all stacks and resources to the appropriate accounts.

The following templates are deployed as part of the solution:
- [Basic Project Template](docs/templates/BASIC_PROJECT_TEMPLATE.md)
- [Dynamic Account Project Template](docs/templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md)
- [BYOC Project Template](docs/templates/BYOC_PROJECT_TEMPLATE.md)

## Getting Started

Expand Down
Loading