Open Development & Education - Evidence Library application

This application uses Kerko to provide a user-friendly search and browsing web interface for a bibliography managed with the Zotero reference manager.

This is a fork of KerkoApp (version 1.0).

About this application

Some things to know:

• Although the structure and some bits of code are similar to the standard KerkoApp, there is no guaranty that this application is compatible with KerkoApp in any way. Since this application is just a thin layer built over Kerko, it is relatively small and can follow its own destiny.
• Configuration is generally done in config.toml, except for secret keys that should not be stored in source control and thus go to .secrets.toml. Please refer to Kerko's documentation on configuration for details.
• More advanced customizations are generally done in kerkoapp/config.py.
• Some of Kerko's templates are overridden (see the kerkoapp/templates/kerkoapp/ directory).
• The custom Sass stylesheet takes advantage of Bootstrap's theming capabilities (see static/src/app/scss/styles.scss).
• Front-end assets such as CSS, JavaScript and icons are bundled with the application. In production, these files are statically served from the static/dist/ directory.
• Similarly to back-end Python packages, front-end dependencies are retrieved during the installation process. While the former go into your Python virtual environment, the latter go to the static/src/vendor/ directory, which should not be stored in the code repository.
• If some source assets from static/src/ are modified, you have to run the build process, which generates the content of static/dist/. You normally push the resulting files from static/dist/ to the code repository, so that the built assets can be used on the production server.

Developing the application

Making changes to this application requires a Python development environment. Once the changes are tested in that environment, they can be pushed to the Git repository and deployed on the production server (see the Deploying the application section).

Installing the application locally

Pre-requisites:

• A Python virtual environment using the same Python version as the production server (Python 3.7).
• Install Node.js (recommended version: 10.x or later). Node.js provides npm, a package manager that is required to install some of the application's front-end dependencies.

Steps:

1. With the virtual environment active, install the software by running the following shell commands:

   git clone git@github.com:OpenDevEd/Zotero-kerko-docs.git opendeveddocs
   cd opendeveddocs
   pip install -r requirements/dev.txt
   npm install

   This will install many packages required by Kerko and this application.

2. Copy sample.secrets.toml to .secrets.toml. Open .secrets.toml in a text editor to assign proper values to the variables.

3. Synchronize data from zotero.org:

   flask kerko sync

   For a large bibliography and/or large file attachments, that command may take a while to complete. In production use, that command is usually added to the crontab file for regular execution.

   Note that Kerko provides a few Flask subcommands. To list them all:

   flask kerko --help

   To get details about a given subcommand:

   flask kerko SUBCOMMAND --help

4. Run the application using Flask's built-in server:

   flask run

5. With the server running, open http://localhost:5000/ in your browser to use the application.

Press CTRL+C from the terminal if you wish to stop the server.

Note that Flask's built-in server is not suitable for production as it doesn't scale well, but is is perfectly adequate for development.

Upgrading Python dependencies

There are two types of Python dependencies: (1) run dependencies, which are required to run the application; (2) dev dependencies, which are required to build the application. Those are specified in requirements/run.in and requirements/dev.in respectively. To ensure reproducible results, exact package versions are pinned into requirements/run.txt and requirements/dev.txt.

With your virtual environment active, to upgrade a package PACKAGE to its latest version and synchronize all installed dependencies:

pip-compile --upgrade-package PACKAGE --allow-unsafe --resolver=backtracking --output-file requirements/run.in
pip-compile --upgrade-package PACKAGE --allow-unsafe --resolver=backtracking --output-file requirements/dev.in
pip-sync requirements/dev.txt

After adequate testing, the updated requirements/{dev,run}.{in,txt} files can be pushed to the repository for later deployment.

Upgrading front-end dependencies

There are two types of front-end dependencies: (1) asset dependencies, parts of which are ingested by the build process and packaged into bundles, e.g., Bootstrap, jQuery; (2) dev dependencies, which are tools required by the build process, e.g., clean-css-cli, postcss-cli.

To upgrade an asset dependency, manually edit the package's specification in frontendDependencies section of package.json, then run the following command:

npm install

To upgrade a dev dependency PACKAGE to a version VERSION, run the following command:

npm install PACKAGE@VERSION --save-dev

After a build (see the Building the assets section below) and adequate testing, the updated package.json and package-lock.json files can be pushed to the repository.

Upgrading or changing the configuration

Kerko can be upgraded like regular Python packages (see upgrading Python dependencies). Make sure to check the following sources before attempting an upgrade:

• Kerko changelog
• KerkoApp changelog
• Kerko upgrading documentation

Making any change to a configuration file requires that you at least restart the application afterwards for the change to become effective.

Moreover, some parameters have an effect on the structure of the cache or the search index that Kerko depends on. Changing this kind of parameter may require that you rebuild either. Refer to the documentation of the parameter to check if specific actions need to be taken after a change.

For example, to rebuild the search index:

flask kerko clean index
flask kerko sync

Building the assets

If some front-end dependencies have been upgraded or if you have manually edited a Sass stylesheet (from static/src/app/scss/), a rebuild of the assets is required. From the application's directory (the one that contains wsgi.py), and with your virtual environment active:

export PATH=`pwd`/node_modules/.bin:${PATH}
flask --debug assets build

If you're happy with the results, build the minified assets for production use:

flask assets build

Then push the updated files from the static/dist/ directory to the repository for later deployment.

Note: Never manually edit the files in static/dist/css/ or static/dist/js/; any change will be overwritten by the build process.

Managing the translations

This application maintains its own translations of Kerko's messages. These can be updated from a local install of Kerko, i.e., installed with pip install -e kerko==VERSION. All of the commands below should be run from Kerko's directory and specify the application's translations directory (here indicated by YOUR_TRANSLATIONS_DIR).

Create a new PO file (for a new locale) based on the POT file. Replace YOUR_LOCALE with the appropriate language code, e.g., en_GB:

python setup.py init_catalog --locale YOUR_LOCALE --output-dir YOUR_TRANSLATIONS_DIR

Update an existing PO file based on the POT file:

python setup.py update_catalog --locale YOUR_LOCALE --output-dir YOUR_TRANSLATIONS_DIR

Compile MO files:

python setup.py compile_catalog --directory YOUR_TRANSLATIONS_DIR

Note: the compiled MO files are pushed to the repository to ease deployments.

Deploying the application

There are two types of deployment: the initial installation or the deployment of changes to an existing installation.

Installing the application on Gandi

The following procedure has to be performed only once.

1. Create a new Simple Hosting instance.

2. Add SSH key(s) to the instance.

3. Link domain to the instance.

4. Add Git remote to the local Git repository, e.g.:

   git remote add gandi ssh+git://{instance_id}@git.{datacenter_id}.gpaas.net/default.git
   git push gandi master

   (Replace {instance_id} and {datacenter_id} with the appropriate values)

5. Deploy to the instance, e.g.:

   ssh {instance_id}@git.{datacenter_id}.gpaas.net 'deploy default.git'

6. Activate the Emergency Console (SSH)

7. SSH into the instance (requires instance password, SSH keys not supported):

   ssh {instance_id}@console.{datacenter_id}.gpaas.net

8. In the instance, create the .secrets.toml file in /srv/data/web/vhosts/default/. See step 2 of Installing the application locally for details.

9. Synchronize data from zotero.org:

   cd /srv/data/web/vhosts/default
   /srv/data/web/vhosts/default/local/bin/flask kerko sync

10. In a browser, check that the site works.

11. Add scheduled task to /srv/data/etc/cron/anacrontab, e.g.:

    @daily 0 kerkosync cd /srv/data/web/vhosts/default && /srv/data/web/vhosts/default/local/bin/flask kerko sync
    

Deploying changes to the existing installation on Gandi

The following procedure is necessary to deploy changes to this application into production.

1. Once all required changes have been implemented, built, and tested in the local environment, tag and push the new version to the repository, e.g.:

   git tag prod-`date -u +%Y%m%d-%H%M`
   git push gandi master && git push --tags

   If something goes wrong once in production, it will be easy to revert to the previously tagged version.

2. Deploy to the instance, e.g.:

   ssh {instance_id}@git.{datacenter_id}.gpaas.net 'deploy default.git'

3. If changes require the search index to be rebuilt, SSH to the instance and run the following commands:

   cd /srv/data/web/vhosts/default
   /srv/data/web/vhosts/default/local/bin/flask kerko clean index
   /srv/data/web/vhosts/default/local/bin/flask kerko sync