DeployEx

Who supervises the supervisor (of your application)?

DeployEx is a lightweight tool designed for managing deployments for Beam applications (Elixir, Gleam and Erlang) without relying on additional deployment tools like Docker or Kubernetes. Its primary goal is to utilize the release package for executing full deployments or hot-upgrades, depending on the package's content, while leveraging OTP distribution for monitoring and data extraction.

DeployEx acts as a central deployment runner, gathering crucial deployment data such as the current version and release package contents. The content of the release package enables it to run for a full deployment or a hot-upgrade. Meanwhile, on the development front, your CI/CD pipeline takes charge of crafting and updating packages for the target release. This integration ensures that DeployEx is always equipped with the latest packages, ready to facilitate deployments.

DeployEx is currently used by:

• Calori Web Server for Elixir applications and you can check it at homepage.
• Cochito Web Server for Gleam applications and you can check it at homepage.
• Snake Game with Cowboy for Erlang applications and you can check it at homepage.

Upon deployment, the following dashboard becomes available, offering access to logs for both DeployEx and monitored applications, along with an IEX and/or erl terminal."

🔉 Features

• Monitors multiple Beam application instances (Elixir/Erlang/Gleam) and automatically restarts them if they crash for any reason.
• Includes a backoff delay restart mechanism to prevent excessive restarts.
• Performs full deployments based solely on the release files generated by:
  • mix release for Elixir.
  • gleam export for Gleam.
  • rebar3 as prod tar for Erlang.
• Supports hot code reloading for:
  • Elixir applications using the Jellyfish library.
  • Erlang applications using the rebar3_appup_plugin plugin.
• Supports the following cloud providers:
  • Amazon Web Services (AWS)
  • Google Cloud Provisioning (GCP)
• Provides rollback functionality if a monitored app version remains unstable for 10 minutes.
• Rolled-back monitored app versions are ghosted, preventing their redeployment.
• Ensures all instances remain connected to the OTP distribution, including DeployEx itself.
• Supports OTP distribution with mutual TLS (mTLS) for secure monitoring of apps and DeployEx.
• Provides the ability to run pre-commans prior deployments for Database migrations or any other eval command.
• Provides a friendly UI that only authenticated users can access.
• Allows setting a previously configured version in the UI, enabling DeployEx to enforce deployment of a specific version.
• Supports individual application restarts via the UI, including DeployEx itself.
• Allows access to current log files (stdout and stderr) for both monitored apps and DeployEx.
• Provides access to the shell:
  • IEx shell for monitored Elixir apps and DeployEx.
  • Erlang shell for monitored Gleam/Erlang apps.
• Provides installer script to be used with ubuntu hosts.
• Provides status information per instance:
  • OTP connectivity
  • Version history
  • Last deployment status
  • Number of crash restarts
  • Number of forced restarts
• And much more to come ...

Note

All examples and deployments in this project use NGINX as a reverse proxy and load balancer. However, DeployEx does not depend on NGINX; it is used here purely for convenience.

⚠️ Next steps

Version 0.3.0

• 🚧 Add Gleam support.
• 🚧 Add Erlang support.
• 🚧 Add Erlang hot upgrade support.

Version 0.4.0

• 🚧 Add telemetry support for DeployEx to capture metrics and telemetry via OTP distribution.
• 💤 Lazy deployments for Phoenix apps (Delay Endpoint start to allow fast switch for full deployments)
• 💤 Integrate CPU utilization monitoring from the OTP distribution.
• 💤 Continuous improvement in UI design.

No version defined yet

• 💤 Health checks via OTP distribution
• 💤 Secrets from Environment variables to allow installation on servers with no cloud provider
• 💤 Orchestrate distributed databasses like Khepri
• 💤 Enhance installer to become an elixir app capable of hotupgrade DeployEx

📁 Getting Started

Running the application

You can kickstart the setup with the following commands, the default number of replicas is 3:

mix deps.get
iex --sname deployex --cookie cookie -S mix phx.server
[info] Initialising deployment server
[info] Running DeployexWeb.Endpoint with Bandit 1.5.7 at 127.0.0.1:5001 (http)
[info] Access DeployexWeb.Endpoint at http://localhost:5001
[watch] build finished, watching for changes...
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Interactive Elixir (1.16.0) - press Ctrl+C to exit (type h() ENTER for help)

Rebuilding...

Done in 390ms.
[error] Invalid version map at: /tmp/myphoenixapp/versions/myphoenixapp/local/current.json reason: enoent

Now you can visit localhost:5001 from your browser and enter the credentials for the admin user, username: admin password: deployex. You should expect the following dashboard:

Note

The error message in the CLI is due to no monitored app is available to be deployed. If you want to proceed for a local test, follow the steps at Running DeployEx and Monitored app locally. Also, it is important to note that the distribution will be required so this is the reason to add -sname deployex in the command

How DeployEx handles monitored application Version/Release

The DeployEx app expects a current.json file to be available, which contains version, hash information and any pre-command. This file is mandatory for full deployment and hot upgrades.

Version file (current.json)

Expected location in the release folder:

# production path
./{bucket}/versions/{monitored_app}/{env}/current.json
# local test path
/tmp/{monitored_app}/versions/{monitored_app}/{env}/current.json

Expected JSON format for current.json:

{
  "version": "1.0.0",
  "pre_commands": [ "eval MyApp.Migrator.create", "eval MyApp.Migrator.migrate" ], # optional field
  "hash": "local"
}

Once the file is captured, the deployment will start if no app is running or if the current app is running with a version that differs from the current.json file.

Release package

Expected location in the release folder:

# production path
./{bucket}/dist/{monitored_app}/{monitored_app}-{version}.tar.gz
# local test path
/tmp/{monitored_app}/dist/{monitored_app}/{monitored_app}-{version}.tar.gz

🛠️ Production Information

Environment Variables

DeployEx application typically requires several environment variables to be defined for proper operation. Ensure that you have the following environment variables set when running in production where the ones that have a default value available are not required:

ENV NAME	EXAMPLE	SOURCE	DEFAULT	DESCRIPTION
DEPLOYEX_SECRET_KEY_BASE	42otsNl...Fpq3dIJ02	aws secrets	-/-	secret key used for encryption
DEPLOYEX_ERLANG_COOKIE	cookie	aws secrets	-/-	erlang cookie
DEPLOYEX_ADMIN_HASHED_PASSWORD	$2b$1...5PAYTZjNQ42ASi	aws secrets	-/-	Hashed admin password for authentication
DEPLOYEX_MONITORED_APP_NAME	myphoenixapp	system ENV	-/-	Monitored app name
DEPLOYEX_MONITORED_APP_LANG	elixir, gleam or erlang	system ENV	-/-	Monitored app language
DEPLOYEX_CLOUD_ENVIRONMENT	prod	system ENV	-/-	cloud env name
AWS_REGION	us-east2	system ENV	-/-	the aws region
GOOGLE_APPLICATION_CREDENTIALS	/path/to/file.json	system ENV	-/-	the google application credentials path
DEPLOYEX_PHX_HOST	example.com	system ENV	-/-	The hostname for your application
DEPLOYEX_PHX_PORT	5001	system ENV	5001	The port on which the application will run
DEPLOYEX_PHX_SERVER	true	system ENV	true	enable/disable server
DEPLOYEX_RELEASE_ADAPTER	s3 or gcp-storage	system ENV	-/-	release adapter type
DEPLOYEX_RELEASE_BUCKET	myphoenixapp-prod-distribution	system ENV	-/-	release distribution bucket name
DEPLOYEX_SECRETS_ADAPTER	aws or gcp	system ENV	-/-	release adapter type
DEPLOYEX_SECRETS_PATH	deployex-myphoenixapp-prod-secrets	system ENV	-/-	secret path to be retrieved from
DEPLOYEX_MONITORED_APP_PORT	4000	system ENV	4000	the initial port for starting the monitored apps
DEPLOYEX_MONITORED_REPLICAS	2	system ENV	3	Number of replicas to monitor
DEPLOYEX_DEPLOY_TIMEOUT_ROLLBACK_MS	600000	system ENV	600000	The maximum time allowed for attempting a deployment before considering the version as non-deployable and rolling back
DEPLOYEX_DEPLOY_SCHEDULE_INTERVAL_MS	5000	system ENV	5000	Periodic checking for new deployments

For local testing, these variables are not expected or set to default values.

☁️ Cloud Providers

DeployEx offers a comprehensive set of Terraform examples for programmatically deploying in AWS and GCP, including detailed step-by-step setup instructions:

• AWS deployment for Elixir
• GCP deployment for Elixir
• AWS deployment for Gleam

Installation

If you intend to install DeployEx directly on an Ubuntu server, you can utilize the installer script provided in the release package. For an example of monitored app, please see the setup for the Calori Web Server - AWS/Calori Web Server - GCP. The installer script requires a JSON configuration file, an example of which can be found here. This JSON file can also export environment variables specific to the monitored applications.

Currently, the release and installation process supports Ubuntu versions 20.04 and 22.04. However, you have the option to manually compile and install DeployEx on your target system.

Pre-commands (Elixir only)

Your application will likely require database commands, such as migrations. DeployEx handles these through pre-commands specified in current.json under the pre_commands field. These commands will be executed in the order they are listed, before the application starts. If a pre-command is needed and does not require changes to the application itself, using pre-commands in conjunction with hotupgrade is ideal to avoid unnecessary downtime.

🔐 Secrets Requirements

DeployEx uses Secret Manager (AWS or GCP) to fetch its secrets via the config provider. The following environment variable configuration is expected for Secret Manager:

DEPLOYEX_SECRETS_ADAPTER=gcp
DEPLOYEX_SECRETS_PATH=deployex-myapp-prod-secrets

Within the secrets, the following key-value pairs are required:

ENV NAME	EXAMPLE	DESCRIPTION
DEPLOYEX_SECRET_KEY_BASE	42otsNl...Fpq3dIJ02	mix phx.gen.secret
DEPLOYEX_ERLANG_COOKIE	my-cookie	erlang cookie
DEPLOYEX_ADMIN_HASHED_PASSWORD	$2b$1...5PAYTZjNQ42ASi	Bcrypt.hash_pwd_salt("my-pass")

🏠 Running DeployEx locally

• Elixir Applications
• Gleam Applications
• Erlang Applications

🔨 Throubleshooting

Accessing DeployEx logs

# production
tail -f /var/log/deployex/deployex-stdout.log
tail -f /var/log/deployex/deployex-stderr.log
# local test
# not available when running as dev env

Connecting to the DeployEx IEX CLI

export RELEASE_NODE_SUFFIX=""
export RELEASE_COOKIE=cookie
# production
/opt/deployex/bin/deployex remote
# local test
# not available when running as dev env

Accessing monitored app logs

export instance=1
export monitored_app_name=myphoenixapp
# production
tail -f /var/log/${monitored_app_name}/${monitored_app_name}-${instance}-stdout.log
tail -f /var/log/${monitored_app_name}/${monitored_app_name}-${instance}-stderr.log
# local test
tail -f /tmp/${monitored_app_name}/${monitored_app_name}/${monitored_app_name}-${instance}-stdout.log
tail -f /tmp/${monitored_app_name}/${monitored_app_name}/${monitored_app_name}-${instance}-stderr.log

Connecting to the monitored app manually

Elixir

export instance=1
export monitored_app_name=myphoenixapp
export RELEASE_NODE_SUFFIX=-${instance}
export RELEASE_COOKIE=cookie
# production
/var/lib/deployex/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote
# local test
/tmp/deployex/varlib/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote

Gleam

export instance=1
export monitored_app_name=mygleamapp
export hostname=??? # From the local machine
export ssl_options="-proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf" # If enabled
# production
erl -remsh ${monitored_app_name}-${instance}@${hostname} -setcookie cookie ${ssl_options}
# local test
erl -remsh ${monitored_app_name}-${instance}@${hostname} -setcookie cookie ${ssl_options}

Erlang

export instance=1
export monitored_app_name=myerlangapp
export RELX_REPLACE_OS_VARS=true
export RELEASE_NODE=${monitored_app_name}-${instance}
export RELEASE_COOKIE=cookie
export RELEASE_SSL_OPTIONS="-proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf" # If enabled
# production
/var/lib/deployex/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote_console
# local test
/tmp/deployex/varlib/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote_console

❓How DeployEx handles services

DeployEx operates by monitoring applications and versions using folders and files, treating the monitored app as a service:

# test environment
/tmp/deployex/varlib/service/${monitored_app}/${instance}/previous/${monitored_app}
/tmp/deployex/varlib/service/${monitored_app}/${instance}/new/${monitored_app}
/tmp/deployex/varlib/service/${monitored_app}/${instance}/current/${monitored_app}

# production environment
/var/lib/deployex/service/${monitored_app}/${instance}/previous/${monitored_app}
/var/lib/deployex/service/${monitored_app}/${instance}/new/${monitored_app}
/var/lib/deployex/service/${monitored_app}/${instance}/current/${monitored_app}

The deployment process involves several steps to ensure smooth transitions:

Full deployment

1. Download and Unpack the New Version: The new version of the application is downloaded and unpacked into the new service folder, ready for deployment.
2. Check if the release contain a hot-upgrade or full deployment: DeployEx will check the release file received and if it is a full deployment, goes to the step 3 .
3. Stop the Current Application: The currently running application instance is stopped to prepare for the new deployment.
4. Delete the Previous Service Folder: The previous service folder, containing the previous version of the application, is deleted to make space for the new version.
5. Move the Current Service: The current service folder, representing the current version of the application, is moved to the previous service folder. Simultaneously, the new service folder is moved to become the new current service folder.
6. Run pre_commands: Before starting the application, DeployEx will attempt to run any pre_commands (if set) using the current service folder.
7. Start the Application: Finally, the application is started using the version now residing in the current service folder, ensuring that the latest version is active and operational.

By following this process, DeployEx facilitates deployments, ensuring that applications are updated while minimizing downtime.

Hot-upgrades

For this scenario, there will be no moving files/folders since the target is to keep the current service folder updated. The sequence is:

1. Download and Unpack the New Version: The new version of the application is downloaded and unpacked into the new service folder, ready for deployment.
2. Check if the release contain a hot-upgrade or full deployment: DeployEx will check the release file received and if it is a hot-upgrade, goes to the step 3 .
3. Run pre_commands: Before executing the hot-upgrade, DeployEx will attempt to run any pre_commands (if set) using the new service folder.
4. Execute the Hotupgrade checks and verification DeployEx will try to run the hotupgrade sequence and if succeeds, it makes the changes permanent. In any case of failure, it tries to execute a full deployment with the same release file.

🗨️ Getting involved

☎️ Contact us: Feel free to contact me on Linkedin.

©️ Copyright and License

Copyright (c) 2024, Thiago Esteves.

DeployEx source code is licensed under the MIT License.