Welcome to the aws-codepipeline-stepfunctions project on AWS Labs!

How about delegating complex CodePipeline tasks to a proper state machine and keeping your pipeline clean and easy to understand?

This project shows you how to integrate AWS CodePipeline and AWS Step Functions state machines. The integration enables developers to build much simpler CodePipeline actions that perform a single task and to delegate the complexity of dealing with workflow-driven behavior associated with that task to a proper state machine engine. As such, developers will be able to build more intuitive pipelines and still being able to visualize and troubleshoot their pipeline actions in detail by examining the state machine execution logs.

Project structure

• app/ - contains a CloudFormation template that represents the application being deployed via CodePipeline/Step Functions.

• pipeline/codepipeline - CodePipeline resources

• pipeline/statemachines - Step Functions resources including the Deploy state machine

• pipeline/local_modules - required local npm modules

• pipeline/docs - project's documentation artifacts

Required Software

Please install the following software in your local workstation before proceeding:

• Git client

• AWS Command Line Interface (version 1.11.170 or greater)

• Configure your AWS credentials (access keys, AWS region, etc). At a minimum your credentials should allow you to create and manipulate resources associated withe the following AWS services: Amazon S3, AWS CodeCommit, AWS CodePipeline, AWS Lambda, AWS StepFunctions, and AWS IAM.

aws configure

• Install npm

If you plan to develop and contribute to the project it's a good idea to use SAM Local:

• AWS SAM Local

Instructions

Make all scripts executable

From the project's root directory, type:

chmod u+x **/*.sh

Install required NPM modules

From the project's root directory, type:

  cd pipeline/
  install_dependencies.sh

Deploy Resources

• Clone this project in your local workstation (git clone ...)

• If you wish to use your default AWS profile skip this step, otherwise open script file config.sh and enter your profile using variable AWS_PROFILE.

vi pipeline/config.sh

Important: Make sure the AWS profile chosen uses an AWS region that supports AWS Step Functions (check here).

• From the project's root directory run:

  cd pipeline/
  ./bootstrap.sh

The bootstrap script will create all necessary resources including:

• S3 buckets for CodePipeline, Lambda and the State Machine
• Several Lambda functions for CodePipeline and Step Functions
• The CodeCommit repository
• The Step Functions state machine
• The CodePipeline pipeline

Push Code and Trigger CodePipeline

Once the bootstrap script completes, open the AWS Console and check the various resources created for you including a CodeCommit repository, a CodePipeline pipeline, a Step Functions state machine, three S3 buckets, and IAM roles. Note that the CodePipeline pipeline is in a failing state since the CodeCommit repository does not have any code. The next step will be to push code to the CodeCommit repository which will cause the pipeline to start processing the changes.

Let's add code to the repository.

• Using the AWS Console, navigate to the CodeCommit repository created for you

• Click on "Connect" or follow these instructions to set up credentials to your IAM user to allow you to clone the CodeCommit repository (via SSH or HTTPS).

• Back in your local workstation. From the project's root directory, type:

  cd app/
  git init && git add --all
  git commit -m "initial commit"
  git remote add codecommit [YOUR-CODECOMMIT-REPO-CLONE-URL]
  git push codecommit master

This will push code into the CodeCommit repository and trigger the CodePipeline pipeline after a few seconds.

Open the CodePipeline Console and follow the execution of the pipeline. The Source pipeline action will simply load the source code from CodeCommit into an S3 bucket. The Deploy action invokes the StateMachineTriggerLambda Lambda function which, in turn, fetches the state machine input parameters from a file in S3 and triggers the state machine (see figure above for further details). The state machine starts executing while the StateMachineTriggerLambda Lambda sends a continuation token to CodePipeline and terminates. Seconds later, the StateMachineTriggerLambda Lambda is invoked again by CodePipeline. The Lambda will check whether the state machine execution has completed and, if that's the case, it will notify the pipeline that the pipeline action succeeded. If otherwise the state machine has failed, the Lambda will send a failure response to the pipeline action interrupting the pipeline execution. The StateMachineTriggerLambda Lambda fully decouples the pipeline from the state machine.

For further details please read our AWS DevOps Blog post

Contributing

Here are a few details you need to know if you wish to contribute to this project.

Running Locally

Please use AWS SAM Local to run and debug Lambda functions locally. It is really useful and saves a lot of time. A script named run_local.sh is available under pipeline/codepipeline and pipeline/statemachine/deploy for convenience.

In order to run Lambda functions locally, SAM local requires that test events are passed as input to the Lambda functions. Sample test events are provided under test_events/. Important: the provided test events might contain account-specific parameters that need to be adjusted.

Here is an example of how to invoke the CreateStackStateMachineTask Lambda function:

  cd pipeline/statemachines/deploy
  ./run_local.sh CreateStackStateMachineTask create_stack

Note that template state_machine_template.yaml and test event file test_events/event_create_stack.json are referenced within the script.

Modifying Local Modules

If you plan to make changes to local modules (under _pipeline/local_modules) keep in mind that this requires updating the modules that depend on these local modules.

This can be accomplished by running these scrips:

 # pack changes made to local modules
 cd pipeline/local_modules
 ./pack_modules.sh
   
 # update codepipeline local modules references
 cd pipeline/codepipeline
 ./install_modules.sh
   
 # update statemachines/deploy local module references
 cd pipeline/statemachines/deploy
 ./install_modules.sh

Have fun and contribute!