Artifacts upload

[benoit74]

Rationale

• Fix On-demand archiving of temporary files produced by scrapers #913
• Fix what has been found to not be relevant around this change

Remark:

• there are many changes, I suggest to review them commit by commit to gain a better understanding since commits are meant to provide meaning to the change made
• I did not included any upgrade of other frontend or backend dependencies, I prefer to do it in a separate PR (but I did not omit this is needed as well)

Changes

• Always add --keep to zimit offliner command
  • Needed if we want to archive these artifacts (e.g. warc.gz files)
  • We do not mind since the container is deleted almost right after anyway
  • Exposing the parameter in Zimfarm adds no value for the end user and there is significant risk of mistake (very frustrating to restart a recipe with artifacts globs configured ... only to realize after multiple hours you forgot to also activate the --keep setting)
• Update QA tools and fix code accordingly
• Add artifacts functionality
  • Add a artifacts_globs configuration which is a list of strings
  • Modify API and UI to manipulate this new configuration
  • Modify task worker to check if artifacts_globs is configured after scraper has completed, and if so:
    • tar the files matching any of the globs in a single tar file
      • in-tar hierarchy match the original filesystem hierarchy to avoid to have to deal with conflicting file names
    • as files are added to the tar archive, delete them from the filesystem
    • start an upload container to upload the tar file
    • configuration is done through two new environment variables (needed to be set at task request, values are saved in task afterwards to be pushed to the worker): ARTIFACTS_UPLOAD_URI and ARTIFACTS_EXPIRATION
• Enhance checks at recipe creation to avoid inconsistencies between selected offliner and image name
• Enhance recipe config patching to delete keys which can be unset (platform and new artifacts_globs)
• Enhance secrets removal: always remove what looks like an S3 secret in what looks like a URL, in any key of the response, no matter where it is in the structure
• Fix tests around recipe patching:
  • in good patch tests, the assertions made were not checking the configuration content ... which is the main part modified by the patch
  • in bad patch tests fixture, many data was not needed and confusing, patch request was reduced to the strict minimum to make the test fail (i.e. changing a single dict value in test fixture make the test fail because data becomes valid)
  • add lots of comment to ensure we know what we are testing (so that it is easier to fix when an issue in encountered)
• Small cosmetic changes to recipe configuration UI:
  • Enhance descriptions of Docker image name and tag, and do not suggest "latest" as default value (this is not always a good idea, and not our usage anymore in none of our instances)
  • Add missing semicolons

Screenshots

Recipe configuration (view)

Recipe configuration (edit)

New button in task debug screen to download artifacts:

Result tar content:

[rgaudin]

Thank you ; please look at inline suggestions ; doesn't require re-review