Skip to content

Commit

Permalink
ci: merge doc and workflow changes from dev
Browse files Browse the repository at this point in the history
  • Loading branch information
spwoodcock committed Sep 11, 2023
1 parent ccc7820 commit 92f07fd
Show file tree
Hide file tree
Showing 9 changed files with 183 additions and 39 deletions.
57 changes: 57 additions & 0 deletions .github/workflows/ci_img_build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Build CI Img

on:
# Push includes PR merge
push:
branches:
- main
- staging
- development
paths:
# Workflow is triggered only if deps change
- "src/backend/pyproject.toml"
- "src/backend/Dockerfile"
# Allow manual trigger
workflow_dispatch:

env:
REGISTRY: ghcr.io
GIT_BRANCH: ${{ github.ref_name }}

jobs:
build-and-push-images:
runs-on: ubuntu-latest
environment:
name: ${{ github.ref_name }}
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Log in to the Container registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract api version
id: extract_api_version
run: |
cd src/backend
echo "API_VERSION=$(python -c 'from app.__version__ import __version__; print(__version__)')" >> $GITHUB_ENV
- name: Build image
uses: docker/build-push-action@v4
with:
context: src/backend
target: ci
push: true
tags: |
"ghcr.io/hotosm/fmtm/backend:${{ env.API_VERSION }}-ci-${{ github.ref_name }}"
"ghcr.io/hotosm/fmtm/backend:ci-${{ github.ref_name }}"
build-args: |
API_VERSION=${{ env.API_VERSION }}
64 changes: 45 additions & 19 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,43 +20,69 @@ permissions:
jobs:
test:
runs-on: ubuntu-latest
environment:
name: test

container:
image: ghcr.io/hotosm/fmtm/backend:ci
image: ghcr.io/hotosm/fmtm/backend:ci-${{ github.base_ref || github.ref_name }}
env:
FMTM_DB_URL: postgresql+psycopg2://fmtm:fmtm@fmtm-db:5432/fmtm
ODK_CENTRAL_URL: http://localhost:8383
ODK_CENTRAL_USER: odk
ODK_CENTRAL_PASSWD: odk
OSM_CLIENT_ID: test
OSM_CLIENT_SECRET: test
OSM_URL: https://www.openstreetmap.org
OSM_SCOPE: read_prefs
OSM_LOGIN_REDIRECT_URI: http://127.0.0.1:8000/auth/callback/
OSM_SECRET_KEY: test
URL_SCHEME: "http"
FRONTEND_MAIN_URL: "localhost:8080"
FRONTEND_MAP_URL: "localhost:8081"
ODK_CENTRAL_URL: ${{ vars.ODK_CENTRAL_URL }}
ODK_CENTRAL_USER: ${{ vars.ODK_CENTRAL_USER }}
ODK_CENTRAL_PASSWD: ${{ vars.ODK_CENTRAL_PASSWD }}
OSM_CLIENT_ID: ${{ vars.OSM_CLIENT_ID }}
OSM_CLIENT_SECRET: ${{ vars.OSM_CLIENT_SECRET }}
OSM_SECRET_KEY: ${{ vars.OSM_SECRET_KEY }}
FRONTEND_MAIN_URL: ${{ vars.FRONTEND_MAIN_URL }}
FRONTEND_MAP_URL: ${{ vars.FRONTEND_MAP_URL }}
options: --user root

# pytest needs a database
services:
# Label used to access the postgres service container
# Start backend database
fmtm-db:
# Uses dockerhub postgis image
image: postgis/postgis:14-3.3-alpine
# Provide the password, initilized database for pytest user
env:
POSTGRES_PASSWORD: fmtm
POSTGRES_DB: fmtm
POSTGRES_USER: fmtm
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
# Start ODK Central database
central-db:
image: "postgis/postgis:14-3.3-alpine"
env:
POSTGRES_PASSWORD: odk
POSTGRES_DB: odk
POSTGRES_USER: odk
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
# Start ODK Central
central:
image: "ghcr.io/hotosm/fmtm/odkcentral:v2023.2.1"
env:
DOMAIN: local
SYSADMIN_EMAIL: test@hotosm.org
SYSADMIN_PASSWD: odk
HTTPS_PORT: 443
DB_HOST: central-db
DB_USER: odk
DB_PASSWORD: odk
DB_NAME: odk
SENTRY_ORG_SUBDOMAIN: o130137
SENTRY_KEY: 3cf75f54983e473da6bd07daddf0d2ee
SENTRY_PROJECT: 1298632

# Start proxy to access ODK Central
central-proxy:
image: "ghcr.io/hotosm/fmtm/odkcentral-proxy:latest"

steps:
- uses: actions/checkout@v3
- name: Run pytest as appuser
Expand Down
9 changes: 6 additions & 3 deletions .github/workflows/wiki.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@ on:
# Allow manual trigger (workflow_dispatch)
workflow_dispatch:

permissions:
contents: write

env:
USER_TOKEN: ${{ secrets.WIKI_DEPLOY_TOKEN }}
TOKEN: ${{ secrets.GITHUB_TOKEN }}
USER_NAME: svcfmtm
USER_EMAIL: fmtm@hotosm.org
ORG: ${{ github.event.repository.owner.name }}
Expand All @@ -31,7 +34,7 @@ jobs:
git init
git config user.name $USER_NAME
git config user.email $USER_EMAIL
git pull https://$USER_TOKEN@github.com/$ORG/$REPO_NAME.wiki.git
git pull https://x-access-token:$TOKEN@github.com/$ORG/$REPO_NAME.wiki.git
- name: Push content to wiki
run: |
Expand All @@ -40,4 +43,4 @@ jobs:
cd tmp_wiki
git add .
git commit -m "docs: automated wiki update on push"
git push -f --set-upstream https://$USER_TOKEN@github.com/$ORG/$REPO_NAME.wiki.git master
git push -f --set-upstream https://x-access-token:$TOKEN@github.com/$ORG/$REPO_NAME.wiki.git master
18 changes: 10 additions & 8 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
> NOTE: This is an installation guide to quickly get the fmtm app up and running. For a detailed guide on how to install the fmtm app using different methods and contributing, checkout the [docs](./docs)
> NOTE: This is an installation guide to quickly get the fmtm app up and running. For a detailed guide on how to install the fmtm app using different methods and contributing, checkout the [dev docs](https://hotosm.github.io/fmtm/dev/Setup/)
# Table of Contents

Expand Down Expand Up @@ -61,7 +61,7 @@ The FMTM uses OAUTH2 with OSM to authenticate users. To properly configure your

1. [Login to OSM](https://www.openstreetmap.org/login) (_If you do not have an account yet, click the signup button at the top navigation bar to create one_). Click the drop down arrow on the extreme right of the navigation bar and select My Settings.

2. Register your local fmtm backend app to OAuth 2 applications. Put your login redirect url as `http://127.0.0.1:8000/auth/callback/`, For Production replace the URL as production API Url
2. Register your FMTM instance to OAuth 2 applications. Put your login redirect url as `http://127.0.0.1:8080/osmauth/`, For Production replace the URL as production API Url

> Note: `127.0.0.1` is required instead of `localhost` due to OSM restrictions.
Expand Down Expand Up @@ -99,7 +99,7 @@ OSM_CLIENT_ID=`<OSM_CLIENT_ID_FROM_ABOVE>`
OSM_CLIENT_SECRET=`<OSM_CLIENT_SECRET_FROM_ABOVE>`
OSM_URL=https://www.openstreetmap.org
OSM_SCOPE=read_prefs
OSM_LOGIN_REDIRECT_URI=http://127.0.0.1:8000/auth/callback/
OSM_LOGIN_REDIRECT_URI=http://127.0.0.1:8080/osmauth/
OSM_SECRET_KEY=<random_key_for_development>
### Database (optional) ###
Expand Down Expand Up @@ -134,9 +134,13 @@ The easiest way to get up and running is by using the FMTM Docker deployment. Do
The FMTM uses ODK Central to store ODK data.

- By default, the docker setup includes a Central server.
- Add an admin user, with the user (email) and password you included in `.env`:
`docker compose exec central odk-cmd --email YOUREMAIL@ADDRESSHERE.com user-create`
`docker-compose exec central odk-cmd --email YOUREMAIL@ADDRESSHERE.com user-promote`
- The credentials should have been provided in your `.env` file to automatically create a user.
- To create a user manually:

```bash
docker compose exec central odk-cmd --email YOUREMAIL@ADDRESSHERE.com user-create
docker-compose exec central odk-cmd --email YOUREMAIL@ADDRESSHERE.com user-promote
```

> Note: Alternatively, you may use an external Central server and user.
Expand Down Expand Up @@ -225,5 +229,3 @@ This is the easiest way to manage multiple frontends at once.
- **Map:** <http://127.0.0.1:8081>

That's it, you have successfully set up the frontend!!

> NOTE: For a detailed guide, installation tips and other installation methods, check out the [docs folder](./docs)
18 changes: 10 additions & 8 deletions docs/_Sidebar.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,26 @@

[About](https://github.com/hotosm/fmtm/wiki/About)

[Code of Conduct](https://github.com/hotosm/fmtm/wiki/Code-of-Conduct)
[Code of Conduct](https://github.com/hotosm/fmtm/wiki/CODE_OF_CONDUCT)

[User Manual for FMTM](https://github.com/hotosm/fmtm/wiki/User-Manual-For-Project-Managers)

[Contribution](https://github.com/hotosm/fmtm/wiki/Contribution)
[Contribution](https://github.com/hotosm/fmtm/wiki/CONTRIBUTING)

[FAQ](https://github.com/hotosm/fmtm/wiki/FAQ)

## For Developers

[1. Getting Started](https://github.com/hotosm/fmtm/wiki/DEV-1.-Getting-Started)
[1. Getting Started](https://github.com/hotosm/fmtm/wiki/dev/Setup)

[2. Backend](https://github.com/hotosm/fmtm/wiki/DEV-2.-Backend)
[2. Backend](https://github.com/hotosm/fmtm/wiki/dev/Backend)

[3. Frontend](https://github.com/hotosm/fmtm/wiki/DEV-3.-Frontend)
[3. Frontend](https://github.com/hotosm/fmtm/wiki/Dev/Frontend)

[4. Database Tips](https://github.com/hotosm/fmtm/wiki/DEV-4.-Database-Tips)
[4. Database Tips](https://github.com/hotosm/fmtm/wiki/dev/Database-Tips)

[5. Docker Tips](https://github.com/hotosm/fmtm/wiki/DEV-5.-Docker-Tips)
[5. Docker Tips](https://github.com/hotosm/fmtm/wiki/dev/Docker-Tips)

[6. Production Deployment](https://github.com/hotosm/fmtm/wiki/DEV-6.-Production-Deployment)
[6. Production Deployment](https://github.com/hotosm/fmtm/wiki/dev/Production)

[7. Troubleshooting](https://github.com/hotosm/fmtm/wiki/dev/Troubleshooting)
5 changes: 5 additions & 0 deletions docs/css/extra.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
:root {
--md-primary-fg-color: #d73f3f;
--md-primary-fg-color--light: #e27575;
--md-primary-fg-color--dark: #c22929;
}
2 changes: 1 addition & 1 deletion docs/dev/Backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ Creating a new release during development may not always be feasible.
```bash
docker compose \
-f docker-compose.yml \
-f docker-compose.josm.yml \
-f josm/docker-compose.yml \
up -d
```

Expand Down
48 changes: 48 additions & 0 deletions docs/dev/Troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Troubleshooting 🆘

## Running FMTM standalone

- Although it's easiest to use Docker, sometimes it may no be feasible, or not preferred.
- We use a tool called PDM to manage dependencies.
- PDM can run in two modes: venv and PEP582 (`__pypackages__`).
- Be careful when running FMTM you are not accidentally pulling in your system packages.

### Tips

- If a directory `__pypackages__` exists, delete it and attempt to
`pdm install`
again.
- If the `__pypackages__` directory returns, then force using venv instead
`pdm config python.use_venv true`
and remove the directory again.
- Troubleshoot the packages PDM sees with:
`pdm run pip list`
- Check a package can be imported in the PDM-based Python environment:

```bash
pdm run python
import fastapi
```

If you receive errors such as:

```bash
pydantic.error_wrappers.ValidationError: 3 validation errors for Settings
OSM_URL
field required (type=value_error.missing)
OSM_SCOPE
field required (type=value_error.missing)
OSM_LOGIN_REDIRECT_URI
field required (type=value_error.missing)
```

Then you need to set the env variables on your system.

If you would rather not do this,
an alternative can be to feed them into the pdm command:

```bash
FRONTEND_MAIN_URL="" FRONTEND_MAP_URL="" \
OSM_CLIENT_ID="" OSM_CLIENT_SECRET="" OSM_SECRET_KEY="" \
pdm run uvicorn app.main:api --host 0.0.0.0 --port 8000
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,5 +73,6 @@ nav:
- Production Deployment: dev/Production.md
- Docker Tips: dev/Docker-Tips.md
- Database Tips: dev/Database-Tips.md
- Troubleshooting: dev/Troubleshooting.md
- API: swagger/index.html
- Class Hierarchy: apidocs/html/inherits.html

0 comments on commit 92f07fd

Please sign in to comment.