Skip to content

Latest commit

 

History

History

doc-index-updater

README.md

doc-index-updater

doc-index-updater

Implementation details

SPCs and PILs

For SPCs and PILs, a server called Sentinel sends either a Delete or a Create message. The latter includes details of how to retrieve the file to be created from Sentinel via SFTP.

PARs

For PARs, the PARs web portal allows users to either create or update (triggers a delete followed by a create) PAR files.

  1. a medical writer accesses the PARs portal and enters metadata and supplies a PDF file
  2. the upload form submits the metadata and the file to the doc-index-updater
  3. the doc-index-updater responds to the upload form with a job id for tracking the job status
  4. the doc-index-updater uploads the PDF file to a blob storage container in Azure for temporary storage with a prefix of temp/
  5. the doc-index-updater pushes the blob information, along with the submitted metadata, to the Azure Service Bus "create" queue to be picked up by the doc-index-updater's create_manager service worker
  6. the create_manager service worker retrieves the message from the Azure Service Bus queue, and uses the metadata and temporary blob storage details to add the new PAR PDF to the search service index and permanent blob storage

The new PAR PDF will then be available from the products.mhra.gov.uk website.

The temp/ blob will be removed from the blob storage according the the storage lifecycle management policy

Development how-to

The following guides explain how to get started developing the doc-index-updater.

To build a docker image:

make docker-build

To push an image to Azure container registry (ACR):

az acr login --name mhraproductsnonprodregistry

docker push mhraproductsnonprodregistry.azurecr.io/products/doc-index-updater

To run locally

Run locally by tunneling redis connection to Azure over TLS using stunnel.

  • install stunnel with homebrew:
brew install stunnel
  • start stunnel in a new terminal window:
stunnel stunnel.conf
make

To get SFTP working locally, enable SFTP login on your machine (instructions for a mac).

Then run make set-sftp-keys to pull the public/private keys for development and install them in your home directory .ssh dir. This will also add entries to your .env.overrides file for the necessary environment variables.

Navigate to the ~/.ssh dir and install the public key on your localhost server by running: ssh-copy-id -f -i ./doc_index_updater <YOUR_USERNAME>@localhost This will add the public key to your localhost ~/.ssh/authorized_keys file.

Environment variables

The environment variables needed are listed in .env.example.

Use this to create an .env file reflecting your environment which the make command will read.

via Azure key vault

Environment variables are shared via Azure key vault.

To get them, run make get-env. Note that you'll have to have followed the instructions on connecting to an existing kubernetes cluster before this will work, as it has the instructions on installing the Azure CLI and authenticating to Azure.

via Azure portal

In the event that environment variables change, find environment variables via the Azure portal or from terraform output.

Don't forget to run make set-env to share with the team.

To run the tests

Run all tests:

make test

Run specific tests by passing <arguments> through to cargo test:

make test TEST=<arguments>

Monitoring

There's a dashboard set up in Azure to monitor latency, traffic, errors and saturation.

To find it, go to Shared Dashboards in the Azure Portal. More details about monitoring can be found in the infrastructure dir.

Releasing

To create a new release and deployment to production, create and push a new tag of the form diu.vX.X.X (e.g. diu.v1.3.0), incrementing as required from the most recent version. The doc-index-updater-release workflow will then automate the creation of a new deployment in Github, add the image for the tagged commit to the production container registry and update the image for production in the deployments repo. This will trigger ArgoCD to update the image in production. You can then update the release notes with any useful detail in Github.