deploy-scripts

1. Demo
2. Overview
3. Installation
4. Adding deployment to a project
5. Sample Deployment
6. Deployment Steps
7. Automated Testing
8. Licence
9. Customizing build steps
10. Configuration Variables

Demo

In action

Example of the project in action

Screenshot

Overview

deploy-scripts is a collection of shell scripts to automate the packaging and deployment of projects.

It works similarly to capistrano, but has fewer dependencies.

Apart from the build dependencies of the project you are tying to deploy, it should only need the following dependencies:

- git               # Currently only works with git repos
- /bin/sh           # A Bourne-like shell found on most unix-like systems
- openssh-client    # An SSH client to connect with remote machines where the software will be deployed
- docker            # (Optional) The docker command line tools in case the project uses docker
- docker-compose    # (Optional) The docker-compose tool to build images in case the project uses docker
- kubectl           # (Optional) The kubectl command line tool in case the project works with kubernetes
- aws-cli           # (Optional) The AWS command line tool (version <2) in case the project deploys to AWS ECS

The project is still under heavy development and is being updated to include support for other tools.

As much as possible, any new code added to this project should be able to run in a POSIX shell. Exceptions can be made depending on the project you are deploying (eg. bash is ok to use with rails specific parts because a lot of rails related tools need bash anyway).

The following stacks are currently supported

• Java lib projects and Spring Boot (with Maven Wrapper used for building the jar file)
• Ruby on Rails
• Django
• React JS
• Node JS
• Next JS
• PHP
• actix-web (rust web framework)
• Static HTML

Support has also been added to containerize and deploy projects built on the above stacks with:

• Docker
• Kubernetes
• Amazon ECS

Installation

deploy-scripts can be installed in any one of the following 2 ways

Git checkout

Just make a git checkout of the stable branch to your home directory with the following command.

git clone --single-branch --branch 0.6.0 https://github.com/masalachai/deploy-scripts.git $HOME/.deploy-scripts/0.6.0

Bootstrap script for docker image

Or download a bootstrap script that will use a dockerized version of deploy-scripts to manage your deployments. You can save the script to one of the directories in your PATH environment variable for easy execution

# To just download as a shell script
curl https://raw.githubusercontent.com/masalachai/deploy-scripts/0.6.0/deploy-scripts.sh -o $HOME/deploy.sh

# To download the script and update your PATH variable for easy execution
mkdir -p $HOME/.deploy-scripts \
	&& curl https://raw.githubusercontent.com/masalachai/deploy-scripts/0.6.0/deploy-scripts.sh -o $HOME/.deploy-scripts/deploy \
	&& chmod +x $HOME/.deploy-scripts/deploy \
	&& echo 'PATH="$HOME/.deploy-scripts:$PATH"' >> ~/.profile \
	&& . ~/.profile

Adding deployment to a project

Deployment support can be added to a project with the following commands

Git checkout installation

If you have installed deploy-scripts with the git checkout method mentioned above, you can install the required files to your project directory with the following commands.

cd $HOME/.deploy-scripts/0.6.0/installer

# The installer script usage is
# sh install.sh [project type] [your project directory]
# For example:
# For a Java Maven (mvnw) Project
sh install.sh java /path/to/java/project

# or for a Rails project
sh install.sh rails /path/to/rails/project

# or for a Django project
sh install.sh python /path/to/django/project

# or for an actix-web project
sh install.sh rust /path/to/actix/project

# or for a reactjs project
sh install.sh reactjs /path/to/reactjs/project

# or for a nextjs project
sh install.sh nextjs /path/to/nextjs/project

# or for a node project
sh install.sh node /path/to/node/project

# or for a simple PHP or static HTML site
sh install.sh html /path/to/project

Docker image installation

If you have installed deploy-scripts as a docker image as mentioned above, you can install the required files to your project directory with the following commands.

# The installer command usage is
# sh deploy-scripts.sh --install [project type] [your project directory]
# For example:
# For a Java Maven (mvnw) Project
deploy --install java /path/to/java/project

# or for a Rails project
deploy --install rails /path/to/rails/project

# or for a Django project
deploy --install python /path/to/django/project

# or for an actix-web project
deploy --install rust /path/to/django/project

# or for a reactjs project
deploy --install reactjs /path/to/reactjs/project

# or for a nextjs project
deploy --install nextjs /path/to/nextjs/project

# or for a node project
deploy --install node /path/to/node/project

# or for a simple PHP or static HTML site
deploy --install html /path/to/project

Follow the instructions given by the installer, if any.

Once the configuration files have been added, you can further configure your deployment by modifying the configuration variables in app-config.sh or the environment-specific config.sh.

To start deploying according to your configured settings, go to your project root and run the following command:

# For deploy-scripts installed via git checkout
# Usage: sh deploy/deploy.sh [environment name]
# For example for the environment called 'development'
sh deploy/deploy.sh development

# For deploy-scripts installed via docker
# Usage: sh deploy-scripts.sh [your project directory] [environment name]
# For example, for deploying the environment named 'development'
deploy /my/project development

Sample Deployment

For a step-by-step understanding of how a deployment happens, and how to add deployment support to a project, please check the Sample Django Deployment wiki page.

Deployment Steps

• Once added to a project, deploy-scripts copies over the following files:

- / # project root
  - /deploy	# the directory where deploy-scripts config files are added (/config/deploy-scripts in rails)
    - app-config.sh	# Project-wide varibles for deployment configuration
    - /environments # multiple environments for the project
      - /default  # an example environment called default, this can be replaced with your own project environments
        - config.sh # environment specific config, values in which can override the ones in app-config.sh
        - /assets # files placed here are copied to the server side deployment, dir structure is maintained
#       - /docker # (optional) environment specific docker files, can override project-wide files
#         - Dockerfile # (optional) if present, will override the one in deploy/docker
#         - docker-compose.yml # (optional) if present, will override the one in deploy/docker
#       - /kubernetes # (optional) kubernetes configs
#         - service.yaml # (optional) config for kubernetes service and deployment for the project
#   - /scripts # (optional) Hook scripts that can be used for pre and post build hooks of each deployment step
#     - repo.sh # (optional) example of a pre and post repo checkout hook file
#   - /docker # (optional) project-wide docker files
#     - Dockerfile # (optional) if present, will be used to build the docker image
#     - docker-compose.yml # (optional) if present, will be used to build/start the container

It can also use the commented out optional files that are listed, but will work without them for non-containerized deployments.

All projects must have the following vars defined (in app-config.sh or config.sh) so that the relevant steps are executed.

# The project type. Currently supported options are: java, rails, python, node, reactjs and nextjs
TYPE=rails
# The name that will be used to label the app that is being deployed, commonly the hostname where the service is made available is used
SERVICE_NAME=service.com

A typical deployment then proceeds in a number of steps, as follows. Some steps are optional. The expectation is that each of these steps should support several options which can be combined for different types of deployments. The number and order of steps can be varied and removal or addition of more steps to the build process can be done through Customizing build steps

List of steps

1. Repo Checkout
2. Build (optional)
3. Format (optional)
4. Package
5. Push (optional)
6. Post-push (optional)

Repo Checkout

The desired branch is cloned or checked out into a working directory (deploy/.build/repo by default) and the deployment is done from that checkout.

Currently only git repos are supported.

Variables:

# The repository to clone
REPO=git@github.com:namespace/repo.git
# The branch to deploy. Typically set per environment in the corresponding config.sh
GIT_BRANCH=default

Build

Optional. In this step, the code may be compiled if needed, to produce the files that will be deployed.

Variables:

# The tool that will build the project. It varies by the project type (set by the TYPE variable)
BUILD=mvnw

Format

Optional. The files to be deployed are assembled in the required structure to be sent to the target server. By default it's done in the deploy/.build/package directory.

Variables:

# The tool that will assemble the files for deployment. It varies by the project type (set by the TYPE variable)
FORMAT=rails

Package

The assembled files in deploy/.build/package are then put into a packaged form that can be delivered to the server. The most common form we use is a new git repo, which is pushed to the target server (or another git repo), but it can take options to package it as a container as well

Variables:

# How to package the files to be deployed to be sent to the server
PACKAGE=git

Push

Optional. The packaged files can then be sent to the deployment server in a number of ways. The default one is a bare git repo on the server that will receive the files and run a post-receive hook to deliver it to the deployment directory.

# The method used to push the packaged files
PUSH=git-bare
# The target server (can also be a git repo URL)
DEPLOYMENT_SERVER=service.com
# Credentials used when pushing with git
DEPLOYMENT_SERVER_USER=user
# The SSH port being used in case of a git push
DEPLOYMENT_SERVER_PORT=22

Post-Push

Optional. Sometimes (currently only in the case of kubernetes, ecs, or deploying a docker image from a registry), you need to perform some actions to update the deployment with the new version from the machine you are deploying from. Those can be performed in this step

# The steps that will be executed in the post push stage
PUST_PUSH=kubernetes

Automated Testing

The project contains automated tests to verify that deployments don't break when new changes are made to deploy-scripts.

Since deployments work over ssh, running the tests requires you to add your own SSH public key to the $HOME/.ssh/authorized_keys file in your home directory.

cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Currently, deployments for the following project types can be tested

# Test Java (Spring Boot) deployment
sh tests/java.sh
# Test Ruby on Rails deployment
sh tests/rails.sh
# Test React JS deployment
sh tests/reactjs.sh
# Test Next JS deployment
sh tests/nextjs.sh
# Test Django deployment
sh tests/django.sh
# Test Static HTML and PHP website deployment
sh tests/php.sh
# Test actix-web deployment
sh tests/actix.sh
# Test Node JS deployment
sh tests/node.sh
# Test deployment with dockerization on remote host (dockerizes sample Django project)
sh tests/docker.sh
# Test deployment with push to docker registry and pull from registry to a remote host
sh tests/docker-pull.sh

Licence

The code is distributed under the MIT License, a copy of which is included in the project repository.

Customizing build steps

The steps that are run during a deployment are set by the STEPS variable. Its is a shell variable with a space separated list of step names, whose default value is STEPS="repo build format package push post_push". A normal deployment proceeds along these steps, provided the proper configuration variables are set correctly.

Adding an additional step

Additional step(s) can be added to a deployment by defining shell variables with the BEFORE_ or AFTER_ prefixes in front of existing build step names to insert a step into the list of existing steps.

For example, to add a custom step called test before build, you can add a variable BEFORE_build="test" to config.sh or app-config.sh.

BEFORE_build="test"

And define the logic to be executed during this step inside a ds_exec_step() function inside either the deploy/scripts/steps/test.sh file (for project wide step) or inside deploy/environments/[environment name]/scripts/steps/test.sh (for environment specific step). The environment specific logic, if present, will override the project-wide one.

All the variables defined inside app-config.sh and config.sh are available in this function.

# deploy/scripts/steps/test.sh

ds_exec_step() {
	title 'test step'
	printf "Running tests for $SERVICE_NAME ... "
}

Complex builds and deployments can be configured containing dependencies from other projects, conditional logic while preparing the deployment, etc. with custom steps.

Configuration Variables

Simple deployment configuration is controlled almost entirely through shell variables, which can be defined in the project-wide app-config.sh or the environment-specific config.sh. Environment-specific variables override project wide variables.

Here is a list of variables that are configurable

TYPE

Project type.

Currently allowed values:

• java

• rails

• python

• node

• reactjs

• nextjs

• rust - currently only tested with actix-web

• html - for static HTML sites or simple PHP sites

BUILD

Build type depending on the project type variable.

Currently allowed values:

• mvnw - for java projects

• cargo - for rust projects

• npm - for reactjs and nextjs projects

FORMAT

The format that files should be arranged in for deployment.

Currently allowed values:

• spring-boot - for java projects

• rails - for rails projects

• node - for node projects

• django - for python projects

• reactjs - for reactjs projects

• nextjs - for nextjs projects

• actix-web - for actix-web projects

• static - for static HTML sites or simple PHP sites

FORMAT_INCLUDE

An optional space-separated list of files and directories relative to the project root to be copied over to the deployment files being readied for packaging

REPO

The git repo from which to clone the project.

GIT_BRANCH

The branch which should be used for the deployment. If left unset, it will use the local git branch for deployment

PACKAGE

The format in which to package the deployment files.

Currently allowed values:

• git - default value. The default method is to create a bare git repo on the deployment server and push the deployment files to it and use a post-receive git hook to set up and start the deployed project. The files are packaged as a git repo which is then pushed to the bare repo

• docker - to package the files into a docker image. Requires a Dockerfile and a docker-compose.yml file to be supplied

• zip - uses the zip program to archive the files to be packaged into a .zip archive

PUSH

The method to deliver the deployment to the destination.

Currently allowed values:

• git-bare default value. By default the deployment files are pushed to a bare git repo on the deployment server and a post-recieve hook is used to set up and start the project

• docker - to push the docker image built when PACKAGE=docker to a docker registry

• s3 - to push to AWS S3. Currently only works with PACKAGE=zip. See S3_BUCKET_PATH variable on how to set the target S3 bucket.

POST_PUSH

The step to execute after the deployment files are pushed to the destination.

• docker-pull - Uses the docker-compose.yml file supplied to pull the image and start the container on a remote host

• kubernetes - creates (or updates) a kubernetes service and deployment with the built docker image

• ecs - restarts an existing Amazon ECS task

SERVICE_NAME

The name to identify the service being deployed. For example my-project.

PROJECT_ENVIRONMENT

Deduced from the name of the directory under environments/. No need to explicitly set.

LINKED_DIRS

Space-separated list of persistent directories to symlink the deployment to. Usually used when sharing persistent directories between deployments.

LINKED_FILES

Space-separated list of persistent files to symlink the deployment to. Usually used when sharing persistent files, like configuration or login/access credential files between deployments.

DEPLOYMENT_SERVER

The hostname or IP address of the server to deploy to.

DEPLOYMENT_SERVER_USER

The SSH user to use to connect to the DEPLOYMENT_SERVER for transferring the deployment files. Default value is deploy

DEPLOYMENT_SERVER_PORT

The SSH port on the remote server to deploy to. Default value is 22.

DEPLOYMENT_DIR

The directory on the remote server to which the project must be deployed. The default value is $HOME/sites/$SERVICE_NAME/$PROJECT_ENVIRONMENT.

DS_DIR

The directory under the project directory where deploy-scripts files are maintained. The default value is deploy/. For rails projects, the default value is config/deploy-scripts.

DS_UPDATE

Update deploy-scripts before deploying project. Default value is true.

REPO_TYPE

The type of repo to fetch the code from. Default and only currently supported value is git.

RESTART_COMMAND

The shell command to execute to start the deployed service. Default value is sh deploy/run.sh restart, which uses the bundled run.sh script which can start/stop spring-boot, rails, and django projects.

RELEASE_COUNT

The no of previous releases to keep on the remote server.

Docker Variables

DOCKERIZE

When set to true, it will use the supplied Dockerfile and docker-compose.yml on the remote server to build an image from the deployed files and start a container with it. This is different from when the PACKAGE variable is set to docker, which will build the image on your local system before pushing it to a registry

DOCKER_REGISTRY

When PUSH=docker, the target docker registry it should push to is specified by the DOCKER_REGISTRY variable.

DOCKER_HOME

When PUSH=docker, the directory containing the config.json from which the docker registry login credentials are read.

DOCKER_DELETE_LOCAL_IMAGE

Default value is false. When set to true, the built image will be deleted from the local system after it is pushed to the docker registry. To use cached docker layers for faster image builds, keep this variable set to false.

Kubernetes Variables

KUBERNETES_CRED

When POST_PUSH=kubernetes, the kubernetes secret that contains the credentials to pull docker images from a private registry. If your DOCKER_REGISTRY is private and needs a username and password to access, you must create a kubernetes secret with the credentials and set this variable.

KUBERNETES_HOME

When POST_PUSH=kubernetes, the directory containing the kubernetes cluster configuration yaml files needed to connect to and manage the cluster you are deploying to. Default value is $HOME/.kube.

KUBERNETES_CLUSTER

When POST_PUSH=kubernetes, the identifier for the kubernetes cluster you are deploying to. deploy-scripts will search for a yaml file named with this identifier to use to connect to the cluster. For example, if is variable is set to my-cluster, it will look for a my-cluster.yaml in the KUBERNETES_HOME directory for the cluster configuration.

KUBERNETES_CLUSTER_CONFIG

When POST_PUSH=kubernetes, you can explicitly specify the path of the cluster yaml file. If unspecified it will try to locate the yaml file at $KUBERNETES_HOME/$KUBERNETES_CLUSTER.yaml.

KUBERNETES_NAMESPACE

The kubernetes namespace under which the service should be deployed. If unspecified, the service is deployed under the default kubernetes namespace.

KUBERNETES_INGRESS

When POST_PUSH=kubernetes, the nginx ingress service being used as a load balancer for your kubernetes services. Currently only HTTP(s) load balancing is set up.

KUBERNETES_NGINX_SERVICE_HOST

When POST_PUSH=kubernetes, this can be used to explicitly specify the hostname to use for the service. If unspecified, it will try to use SERVICE_NAME as the hostname for the service.

KUBERNETES_NGINX_SERVICE_PORT

The service port to use for the deployed kubernetes service. Default value is 80.

KUBERNETES_CERT_MANAGER

When POST_PUSH=kubernetes, the certificate manager that has been configured for certificate issue and renewal on your kubernetes cluster, if you want to enable HTTPS for your deployment.

KUBERNETES_TLS

When POST_PUSH=kubernetes, whether to enable HTTPS for your deployment. Default value is false. If set to true, you must also set the KUBERNETES_CERT_MANAGER variable.

KUBERNETES_REPLICAS

When POST_PUSH=kubernetes, the number of replicas to enable for your kubernetes service. Default value is 1.

AWS Variables

ECS_CLUSTER

When POST_PUSH=ecs, the name of your Amazon ECS cluster where your task is running.

ECS_SERVICE

When POST_PUSH=ecs, the name of your Amazon ECS task that you need to restart after the deployment.

ECS_STOP_RUNNING_TASKS

When POST_PUSH=ecs, whether to kill the currently running task before starting a new one. Default value is false

AWS_PROFILE

The value to use for the --profile argument for aws cli. The default value is default

S3_BUCKET_PATH

When PACKAGE=zip, the resulting zip file can be pushed to an S3 bucket, if the proper set of credentials for aws cli are configured in your home directory. A sample path would look like S3_BUCKET_PATH="my-bucket/prefix1/prefix2/etc". The path shouldn't contain the s3:// protocol string.