Skip to content

Commit b69b18c

Browse files
EugeneLightsOnEugeneLightsOn
authored
Automated migrations workflow(optionally SSH, Database URL or API call) (#211)
* Migrations Git Workflow + protect migrate API endpoint * Migrations Git Workflow + protect migrate endpoint * Migrations Git Workflow + protect migrate endpoint * Migrations Git Workflow + protect migrate endpoint * Migrations Git Workflow + protect migrate endpoint * Migrations Git Workflow + protect migrate endpoint black
1 parent 9125660 commit b69b18c

File tree

8 files changed

+163
-4
lines changed

8 files changed

+163
-4
lines changed

‎.env-template

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22
NEXT_PUBLIC_API_HOSTNAME=http://localhost:8000
33
DATABASE_URL=postgresql+psycopg2://postgres:postgres@db:5432
44

5+
# Migrate access token, used to authenticate requests to the migrate endpoint.
6+
# You can generate it using some random string generator.
7+
MIGRATE_TOKEN=<MIGRATE_TOKEN>
8+
59
# TOOLS
610
PYTHON_INTERPRETER_URL=http://terrarium:8080
711
TAVILY_API_KEY=<API_KEY_HERE>

‎.github/scripts/run_docker_migration.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
cd $SSH_PROJECT_DIR
3+
git pull
4+
make migrate
5+
echo "Migration complete"

‎.github/scripts/run_src_migration.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
cd $SSH_PROJECT_DIR
3+
git pull
4+
source .venv/bin/activate
5+
alembic -c src/backend/alembic.ini upgrade head
6+
echo "Migration complete"

‎.github/workflows/run_alembic_migrations.yml

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: Automated Migrations
2+
# This workflow provides a way to run Alembic migrations using SSH or migration API or database URL.
3+
on:
4+
push:
5+
branches: [ main ]
6+
jobs:
7+
run-alembic-migrations-ssh:
8+
runs-on: ubuntu-latest
9+
environment: production
10+
env:
11+
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
12+
SSH_HOST: ${{secrets.SSH_HOST}}
13+
SSH_PORT: ${{secrets.SSH_PORT}}
14+
SSH_USER_NAME: ${{secrets.SSH_USER_NAME}}
15+
DATABASE_URL: ${{secrets.MIGRATION_DATABASE_URL}}
16+
MIGRATION_API_ENDPOINT: ${{secrets.MIGRATION_API_ENDPOINT}}
17+
MIGRATION_API_TOKEN: ${{secrets.MIGRATION_API_TOKEN}}
18+
# If migration API params are present, use backend API to run migrations.
19+
USE_MIGRATION_API: ${{ (secrets.MIGRATION_API_ENDPOINT != '') && (secrets.MIGRATION_API_TOKEN != '') }}
20+
# If database URL is present, run migrations on DB connection.
21+
# Please note that the database connection should be open to the IP from where the migrations are being run.
22+
# If you are using a private network, you can set up SSH tunnel to run migrations.
23+
USE_DATABASE_URL: ${{ secrets.MIGRATION_DATABASE_URL != '' }}
24+
# If SSH params are present, use SSH to run migrations.
25+
USE_SSH: ${{ (secrets.SSH_PRIVATE_KEY != '') && (secrets.SSH_HOST != '') && (secrets.SSH_USER_NAME != '') && (secrets.SSH_PORT != '')}}
26+
27+
steps:
28+
- name: Run Alembic Migrations SSH
29+
if: ${{ (env.USE_SSH == 'true' && env.USE_MIGRATION_API == 'false' && env.USE_DATABASE_URL == 'false') }}
30+
run: |
31+
echo "Running migrations using SSH"
32+
echo "$SSH_PRIVATE_KEY" > private_key && chmod 600 private_key
33+
ssh -o StrictHostKeyChecking=no -i private_key ${SSH_USER_NAME}@${SSH_HOST} -p ${SSH_PORT} '
34+
export SSH_PROJECT_DIR=~/cohere-toolkit && # Set the project directory here
35+
sh $SSH_PROJECT_DIR/.github/scripts/run_docker_migration.sh && # Delete this command if your toolkit is not in a docker container
36+
sh $SSH_PROJECT_DIR/.github/scripts/run_src_migration.sh # Delete this command if your toolkit is in a docker container
37+
'
38+
- name: Run Alembic Migrations API
39+
if: ${{ (env.USE_SSH == 'false' && env.USE_MIGRATION_API == 'true' && env.USE_DATABASE_URL == 'false') }}
40+
run: |
41+
echo "Running migrations using the migration API"
42+
curl -H "Authorization: Bearer $MIGRATION_API_TOKEN" -X POST $MIGRATION_API_ENDPOINT
43+
- name: Checkout repository
44+
if: ${{(env.USE_SSH == 'false' && env.USE_MIGRATION_API == 'false' && env.USE_DATABASE_URL == 'true') || (env.USE_SSH == 'true' && env.USE_MIGRATION_API == 'true' && env.USE_DATABASE_URL == 'true') }}
45+
uses: actions/checkout@v4
46+
- name: Set up Python 3.11
47+
if: ${{(env.USE_SSH == 'false' && env.USE_MIGRATION_API == 'false' && env.USE_DATABASE_URL == 'true') || (env.USE_SSH == 'true' && env.USE_MIGRATION_API == 'true' && env.USE_DATABASE_URL == 'true') }}
48+
uses: actions/setup-python@v5
49+
with:
50+
python-version: "3.11"
51+
- name: Run Alembic Migrations Database
52+
if: ${{(env.USE_SSH == 'false' && env.USE_MIGRATION_API == 'false' && env.USE_DATABASE_URL == 'true') || (env.USE_SSH == 'true' && env.USE_MIGRATION_API == 'true' && env.USE_DATABASE_URL == 'true') }}
53+
run: |
54+
echo "Running migrations using the database URL"
55+
pip install poetry==1.7.1
56+
export POETRY_VIRTUALENVS_IN_PROJECT=true
57+
poetry install
58+
echo "DATABASE_URL=$DATABASE_URL" > .env
59+
source .venv/bin/activate
60+
alembic -c src/backend/alembic.ini upgrade head
61+
- name: Fail Migration
62+
if: ${{ (env.USE_SSH == 'false' && env.USE_MIGRATION_API == 'false' && env.USE_DATABASE_URL == 'false') }}
63+
run: |
64+
echo "Migration failed. Please provide either SSH, Migration API or Database URL environment variables. See the README for more information."

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Toolkit is a collection of prebuilt components enabling users to quickly build a
1313
- [How to add tools](/docs/custom_tool_guides/tool_guide.md)
1414
- [How to add authentication](/docs/auth_guide.md)
1515
- [How to deploy toolkit services](/docs/service_deployments.md)
16+
- [How to set up Github Actions for automated DB migrations](/docs/github_migrations_action.md)
1617
- [How to customize the theme](/docs/theming.md)
1718
- [How to contribute](#contributing)
1819
- [Try Cohere's Command Showcase](https://coral.cohere.com/)

‎docs/github_migrations_action.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Github Actions for Automated DB Migrations
2+
3+
This guide will help you set up a Github Action to automatically run database migrations when you push changes to your repository.
4+
5+
## Prerequisites
6+
To set up the Github Action, you will need to have the following:
7+
- access with admin permissions to the repository you want to set up the action for
8+
- a database that you want to run migrations on
9+
10+
## Steps
11+
This guide will walk you through the steps to set up the Github Action for automated database migrations.
12+
We will provide several types of migrations execution, including:
13+
- running migrations using SSH access to a remote server
14+
- running migrations on a remote database using connection strings
15+
- running migrations on a remote database using Cohere Toolkit's API migrations endpoint
16+
For each type of migration execution we need to setup Github environment and secrets.
17+
To seting up the Github environment and secrets, follow the steps below:
18+
- navigate to your repository on Github
19+
- click on the `Settings` tab
20+
- click on the `Environments` menu on the left
21+
- click on the `New environment` button or select an existing environment(we will use the `Production` environment in this guide)
22+
- fill in the environment name and click on the `Configure environment` button
23+
- optionally set rules for the environment, for example, you can set the `Production` environment to be available only for the `main` branch
24+
- click on the `Add secret` button
25+
26+
### Running Migrations using SSH access to a remote server
27+
To run migrations using SSH access to a remote server, you will need to set up the following secrets
28+
- `SSH_PRIVATE_KEY` - the private key to use for SSH authentication
29+
- `SSH_HOST` - the hostname or IP address of the remote server
30+
- `SSH_PORT` - the port to use for SSH authentication
31+
- `SSH_USER_NAME` - the username to use for SSH authentication
32+
The sample Github Action workflow file is located at `.github/workflows/run_alembic_migrations.yml`
33+
Set the `SSH_PROJECT_DIR` variable to the path to the project directory on the remote server. See comments in the `run_alembic_migrations.yml` file for more details.
34+
The action scripts are located at `.github/scripts` directory. The `run_docker_migration.sh` script is used
35+
to run the migrations using Docker. The `run_src_migration.sh` script is used to run the migrations using the source code.
36+
If the Toolkit is deployed on a remote server using Docker, please remove `sh $SSH_PROJECT_DIR/.github/scripts/run_src_migration.sh` string from the `run_alembic_migrations.yml`
37+
If the Toolkit is deployed on a remote server using the source code, please remove `sh $SSH_PROJECT_DIR/.github/scripts/run_docker_migration.sh` string from the `run_alembic_migrations.yml`
38+
Please note that this is a basic example and you may need to adjust the scripts to fit your specific use case.
39+
40+
### Running Migrations on a remote database using connection strings
41+
To run migrations on a remote database using connection strings, you will need to set up the following secrets
42+
In this example, we will use a PostgreSQL database available from the internet. If you are using a private database, you may need to set up a VPN or SSH tunnel to access the database.
43+
- `MIGRATION_DATABASE_URL` - the connection string to the database. It should be in the format `postgresql+psycopg2://username:password@host:port/database` eg. `postgresql+psycopg2://postgres:postgres@db:5432`
44+
45+
### Running Migrations on a remote database using Cohere Toolkit's API migrations endpoint
46+
To run migrations on a remote database using Cohere Toolkit's API migrations endpoint, you will need to set up the following secrets
47+
- `MIGRATION_API_ENDPOINT` - the URL of the Cohere Toolkit API migrations endpoint, eg. `https://>Your Cohere Toolkit host name<:>API port, 8000 by default</migrate`
48+
- `MIGRATION_API_TOKEN` - the API token to use for authentication with the Cohere Toolkit API migrations endpoint. You can get the API token from the Toolkit's `.env` file
49+
50+
Please note that this is a basic example and you may need to adjust the scripts to fit your specific use case.
51+
If all the secrets above are set up, the remote database migrations type will used as default.
52+
If some variables are not set, the migration type will be selected based on the set variables.
53+
If script cant select the migration type(some variables are not set), the script will exit with an error message.

‎src/backend/config/auth.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
1+
import os
2+
3+
from dotenv import load_dotenv
4+
from fastapi import Depends, HTTPException, status
5+
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
6+
17
from backend.services.auth import BasicAuthentication, GoogleOAuth, OpenIDConnect
28

9+
load_dotenv()
10+
311
# Add Auth strategy classes here to enable them
412
# Ex: [BasicAuthentication]
513
ENABLED_AUTH_STRATEGIES = []
@@ -9,6 +17,20 @@
917
# Ex: {"Basic": BasicAuthentication()}
1018
ENABLED_AUTH_STRATEGY_MAPPING = {cls.NAME: cls() for cls in ENABLED_AUTH_STRATEGIES}
1119

20+
# Token to authorize migration requests
21+
MIGRATE_TOKEN = os.environ.get("MIGRATE_TOKEN", None)
22+
23+
security = HTTPBearer()
24+
25+
26+
def verify_migrate_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
27+
if not MIGRATE_TOKEN or credentials.credentials != MIGRATE_TOKEN:
28+
raise HTTPException(
29+
status_code=status.HTTP_401_UNAUTHORIZED,
30+
detail="Invalid or missing token",
31+
headers={"WWW-Authenticate": "Bearer"},
32+
)
33+
1234

1335
def is_authentication_enabled() -> bool:
1436
"""

‎src/backend/main.py

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,15 @@
55
from alembic.command import upgrade
66
from alembic.config import Config
77
from dotenv import load_dotenv
8-
from fastapi import FastAPI, HTTPException
8+
from fastapi import Depends, FastAPI, HTTPException
99
from fastapi.middleware.cors import CORSMiddleware
1010
from starlette.middleware.sessions import SessionMiddleware
1111

12-
from backend.config.auth import get_auth_strategy_endpoints, is_authentication_enabled
12+
from backend.config.auth import (
13+
get_auth_strategy_endpoints,
14+
is_authentication_enabled,
15+
verify_migrate_token,
16+
)
1317
from backend.config.routers import ROUTER_DEPENDENCIES
1418
from backend.routers.agent import router as agent_router
1519
from backend.routers.auth import router as auth_router
@@ -95,7 +99,7 @@ async def health():
9599
return {"status": "OK"}
96100

97101

98-
@app.post("/migrate")
102+
@app.post("/migrate", dependencies=[Depends(verify_migrate_token)])
99103
async def apply_migrations():
100104
"""
101105
Applies Alembic migrations - useful for serverless applications
@@ -108,4 +112,4 @@ async def apply_migrations():
108112
status_code=500, detail=f"Error while applying Alembic migrations: {str(e)}"
109113
)
110114

111-
return {"status": "Done"}
115+
return {"status": "Migration successful"}

0 commit comments

Comments
 (0)