Merge pull request #255 from ReadAlongs/dev.ej/pyproject.toml

Perform a complete environment refresh for Studio install, build, deploy

Author: joanise

.github/workflows/matrix-tests.yml

@@ -23,7 +23,6 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Install Python dependencies
         run: |
-          pip install wheel
-          pip install -r requirements.txt -r requirements.dev.txt -r requirements.ci.txt -e .
+          pip install -e .[all]
       - name: Run tests
         run: cd test && python run.py prod

.github/workflows/tests.yml

@@ -24,13 +24,8 @@ jobs:
         run: |
           # Keep pip up to date
           python -m pip install --upgrade pip
-          # Some dependencies are built using wheel
-          pip install wheel
           # Install all Python dependencies in just one pip call, including Studio itself
-          pip install -r requirements.txt \
-                      -r requirements.dev.txt \
-                      -r requirements.ci.txt \
-                      -e .
+          pip install -e .[all]
 
       - name: Run tests
         run: |
@@ -40,22 +35,16 @@ jobs:
           coverage combine
           coverage xml
 
-      - name: Exercise the Heroku Procfile
-        run: |
-          cmd=$(grep web: Procfile | cut -d' ' -f2-)
-          echo "Running $cmd"
-          $cmd &
-          curl --retry 20 --retry-delay 1 --retry-all-errors http://127.0.0.1:8000/api/v1/docs | grep SwaggerUIBundle
-          curl http://127.0.0.1:8000/api/v1/langs | grep Cree
-          kill %1
-
       - name: Make sure pre-commit hooks pass
         uses: pre-commit/action@v3.0.1
 
-      - name: Minimalist license check
-        run: |
-          # Legal check: make sure we don't have or introduce GPL dependencies
-          if pip-licenses | grep -v 'Artistic License' | grep -v LGPL | grep GNU; then echo 'Please avoid introducing *GPL dependencies'; false; fi
+      - name: Automated license checking
+        # Make sure we don't have or introduce dependencies with incompatible licenses.
+        # Exclusions where auto detection and analysis do not work:
+        #  - pympi-ling is MIT, but not auto-detected. https://github.com/dopefishh/pympi
+        #  - text-unidecode is Artistic License or GPL (at our choice): MIT is
+        #    compatible with the former. https://github.com/kmike/text-unidecode/
+        run: licensecheck --zero --ignore-packages pympi-ling text-unidecode
 
       - uses: codecov/codecov-action@v4
         with:
@@ -107,11 +96,7 @@ jobs:
       - name: Install Python dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install wheel
-          pip install -r requirements.txt `
-                      -r requirements.dev.txt `
-                      -r requirements.ci.txt `
-                      -e .
+          pip install -e .[all]
 
       - name: Run tests on Windows
         run: cd test && python run.py prod
@@ -128,3 +113,36 @@ jobs:
           echo Reference ====
           cat test/data/cs-ref.readalong
           if (diff (cat cs.readalong) (cat test/data/cs-ref.readalong)) { throw "Output did not match reference" }
+
+  test-heroku-env:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Read the Heroku run time env and cmd
+        run: |
+          echo "PYTHON_VERSION=$(cat runtime.txt | sed 's/python-//')" >> $GITHUB_ENV
+          echo "RUNTIME_CMD=$(cat Procfile | grep web: | sed 's/web: *//')" >> $GITHUB_ENV
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: "pip"
+
+      - name: Install Python dependencies the way Heroku would
+        run: |
+          pip install -r requirements.txt
+          bin/post_compile
+
+      - name: Launch the API via the Heroku Procfile
+        run: |
+          ${{ env.RUNTIME_CMD }} &
+          curl --retry 20 --retry-delay 1 --retry-all-errors http://127.0.0.1:8000/api/v1/docs | grep SwaggerUIBundle
+          curl http://127.0.0.1:8000/api/v1/langs | grep Cree
+          kill %1
+
+      - name: unit test the web API
+        run: |
+          pip install -e .[dev]
+          python test/test_web_api.py

Contributing.md

@@ -16,7 +16,7 @@ commits.
 Run these commands in each of your sandboxes to enable our pre-commit hooks and commitlint:
 
 ```sh
-pip install -r requirements.dev.txt
+pip install -e .[dev]
 pre-commit install
 gitlint install-hook
 ```
@@ -49,10 +49,10 @@ All the pre-commit hooks are executed using a tool called
 hooks each time you try to commit anything in this repo.
 
 We've listed all the developper dependencies for the project in
-[requirements.dev.txt](requirements.dev.txt) to make them easy to install:
+[pyproject.toml](pyproject.toml) to make them easy to install:
 
 ```sh
-pip install -r requirements.dev.txt
+pip install -e .[dev]
 pre-commit install
 ```
 
@@ -114,7 +114,7 @@ commit-msg hook in Git.
 Run this command in your g2p sandbox to install and enable the commit-msg hook:
 
 ```sh
-pip install -r requirements/requirements.dev.txt
+pip install -e .[dev]
 gitlint install-hook
 ```
 

MANIFEST.in

@@ -236,7 +236,7 @@ This repo follows the [Contributor Covenant](http://contributor-covenant.org/ver
 You can install our standard Git hooks by running these commands in your sandbox:
 
 ```sh
-pip install -r requirements.dev.txt
+pip install -e .[dev]
 pre-commit install
 gitlint install-hook
 ```

README.md

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+# Heroku expects to run your app in place, but doesn't actually do this for you
+pip install -e .

bin/post_compile

@@ -4,4 +4,3 @@ mkdocs-material>=9.5.27
 mkdocs-autorefs>=1.0.1
 mkdocstrings[python]>=0.25.1
 mike>=2.1.1
--r ../requirements.min.txt

docs/requirements.txt

@@ -0,0 +1,113 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "readalongs"
+dynamic = ["version"]
+requires-python = ">= 3.8"
+description = "ReadAlong Studio: Audiobook alignment for Indigenous languages"
+readme = "README.md"
+authors = [
+    { name = "Aidan Pine", email = "Aidan.Pine@nrc-cnrc.gc.ca" },
+    { name = "David Huggins-Daines", email = "dhd@ecolingui.ca" },
+    { name = "Eric Joanis", email = "Eric.Joanis@nrc-cnrc.gc.ca" },
+    { name = "Patrick Littell", email = "Patrick.Littell@nrc-cnrc.gc.ca" },
+    { name = "Delasie Torkornoo", email = "Delasie.Torkornoo@nrc-cnrc.gc.ca" },
+]
+maintainers = [
+    { name = "Eric Joanis", email = "Eric.Joanis@nrc-cnrc.gc.ca" },
+    { name = "Aidan Pine", email = "Aidan.Pine@nrc-cnrc.gc.ca" },
+    { name = "David Huggins-Daines", email = "dhd@ecolingui.ca" },
+]
+
+classifiers = [
+    "Intended Audience :: Education",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Multimedia :: Sound/Audio :: Speech",
+    "Typing :: Typed",
+]
+
+dependencies = [
+    "audioop-lts; python_version>='3.13'",
+    "chevron==0.14.0",
+    "click>=8.0.4",
+    "coloredlogs>=10.0",
+    "fastapi>=0.103.0",
+    "g2p>=1.1.20230822, <3",
+    "lxml>=4.9.4",
+    "numpy>=1.20.2",
+    "pydantic>=1.8.2,<3",
+    "pydub>=0.23.1",
+    "pympi-ling>=1.69,<2.0",
+    "python-slugify==5.0.0",
+    "requests>=2.31.0",
+    "soundswallower~=0.6.0",
+    "webvtt-py==0.4.2",
+]
+
+[project.optional-dependencies]
+dev = [
+    "readalongs[api]",
+    "gitlint-core>=0.17.0",
+    "httpx>=0.24.1",
+    "pep440>=0.1.2",
+    "pre-commit>=2.6.0",
+]
+
+api = [
+    # gunivorn is only available on *nix, but it's only required for production deployments
+    "gunicorn>=23.0.0; platform_system != 'Windows'",
+    # uvicorn works on all platforms and is required for both dev and prod deployments
+    "uvicorn>=0.30.1",
+]
+
+ci = [
+    "readalongs[api]",
+    "coverage",
+    "licensecheck",
+]
+
+all = [
+    "readalongs[dev]",
+    "readalongs[ci]",
+]
+
+[project.scripts]
+readalongs = "readalongs.cli:cli"
+
+[tool.hatch.version]
+path = "readalongs/_version.py"
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "/readalongs",
+]
+
+[tool.hatch.envs.prod]
+features = [ "api" ]
+type = "pip-compile"
+python = "3.13"
+lock-filename = "requirements.txt"
+
+[project.urls]
+Homepage = "https://github.com/ReadAlongs/"
+Documentation = "https://readalongs.github.io/Studio/"
+Repository = "https://github.com/ReadAlongs/Studio"
+Issues = "https://github.com/ReadAlongs/Studio/issues"
+Changelog = "https://github.com/ReadAlongs/Studio/releases"
+
+[tool.mypy]
+plugins = ["pydantic.mypy"]
+ignore_missing_imports = true
+
+[tool.isort]
+profile = "black"

pyproject.toml

@@ -20,7 +20,7 @@
 (or in a dashboard) and will depend on your hosting environment.
 
 You can also spin up the API server grade (on Linux, not Windows) with gunicorn:
-    pip install -r requirements.api.txt
+    pip install -e .[api]
     gunicorn -w 4 -k uvicorn.workers.UvicornWorker readalongs.web_api:web_api_app
 
 Once spun up, the documentation and API playground will be visible at
@@ -124,17 +124,22 @@ async def langs() -> List[SupportedLanguage]:
     """Return the list of supported languages and their names as a dict.
 
     Returns:
-        Supported languages as list with language codes and mapping of
-        language code to name, including minimally the key "_" for the
-        default display name (usually, but not always, in English).
-        For example:
+      Supported languages as a list with language codes and a mapping
+      `{ "_": "Default Display Name" }`, where the language's default display name is
+      usually (but not always) in English.
+      For example:
 
         [
-            {"code": "alq", names: { "alq": "Anishinaabemowin", "_": "Algonquin" }},
-            {"code": "atj", names: { "atj": "Nehiromowin", "_": "Atikamekw" }},
-            {"code": "fra", names: { "fra": "Français", "_": "French" }},
+            {"code": "alq", names: { "_": "Algonquin" }},
+            {"code": "atj", names: { "_": "Atikamekw" }},
+            {"code": "fra", names: { "_": "French" }},
             ...
         ]
+
+      In the future, we may add localized names to this mapping with language
+      codes as the keys, e.g.,
+
+        [ {"code": "alq", names: { "alq": "Anishinaabemowin", "_": "Algonquin" }}, ... ]
     """
     langs, lang_names = LANGS
     return [

readalongs/web_api.py

@@ -1,16 +1,27 @@
 Our production Heroku deployment is controlled by the following files:
  - `Procfile`: tells Heroku what command to launch in each Dyno;
  - `runtime.txt`: tells Heroku which run-time engine to use (i.e., which version of Python);
+ - `requirements.txt`: tells Heroku what our production dependencies are;
+ - `bin/post_compile`: Heroku builds run this after doing `pip install -r requirements.txt`.
 
-   Heroku detects Python by default, but `runtime.txt` lets us specify/bump the version as needed;
- - `requirements.txt`: tells Heroku what our production dependencies are.
+Updating dependencies:
+ - Our dependencies are declared in `pyproject.toml`. This is where changes should be made first.
+ - `requirements.txt` is the generated "lock" file that Heroku uses. To update it,
+   run these commands, preferably from a Linux machine to match the Heroku context:
+
+       hatch env remove prod
+       rm requirements.txt
+       hatch env create prod
+
+   It is also possible to edit it manually, e.g., to handle a critical vulnerability report,
+   but an occasional full rebuild is a good idea, to keep things up to date.
 
 Updating `g2p`:
  - By default, `g2p` only gets updated for `readalong-studio` on Heroku when:
-   - we make a new release of `g2p` on PyPI
-   - we update `requirements.min.txt` here to ask for that release
+   - we make a new release of `g2p` on PyPI, **and**
+   - we update `requirements.txt` here to ask for that release
  - Manual override: it is also possible to update g2p to the current commit on the `main` branch using this procedure:
-   - Change `requirements.min.txt` to specify `g2p @ git+https://github.com/roedoejet/g2p@main`.
+   - Change `requirements.txt` to specify `g2p @ git+https://github.com/roedoejet/g2p@main`.
    - Commit that change to `main` in a sandbox connected to Heroku.
    - `git push heroku main`
    - This will force a build which will fetch the main branch of g2p from GitHub.