Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Surface previous dataset versions across URL redirects #783

Merged
merged 5 commits into from
Jan 21, 2024
Merged

Surface previous dataset versions across URL redirects #783

merged 5 commits into from
Jan 21, 2024

Commits on Jan 19, 2024

  1. [resource indexer] save remote inventories 

    I've supported using local manifest+inventory files from the start as it
makes for a much nicer dev experience than fetching from S3 each time.
This adds the ability to easily save a local copy of the latest
inventory data from S3. This both makes getting started much easier and
provides an easy way to update your local data.
    @jameshadfield
    jameshadfield committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    08f8603 View commit details
    Browse the repository at this point in the history

  2. [charon] refactor manifest parsing code 

    This is to allow the resource indexer to import
`convertManifestJsonToAvailableDatasetList` without the side-effects of
running code on import and setting `global.availableDatasets`.

No code changes, just extracting 2 functions into a new file.
    @jameshadfield
    jameshadfield committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    331f33e View commit details
    Browse the repository at this point in the history

  3. [resource indexer] modify resource URLs 

    following the approach the server uses to redirect URLs. For instance,
we change the resource name for "dengue/denv1" to "dengue/denv1/genome"
because the former URL is redirected to the latter.

This allows the server to access historical datasets for the URL
"dengue/denv1/genome" which now include all "dengue/denv1"
("dengue_denv1.json") datasets.

Due to the way the server currently works this will not allow historical
version access for "dengue/denv1" URLs because the instantiation of the
Dataset (Resource) instance happens before the resolve/redirecting
happens, and the "dengue/denv1" dataset is considered invalid because
it's not present in the index _and_ a version descriptor is being
provided.
    @jameshadfield
    jameshadfield committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2f7cfb9 View commit details
    Browse the repository at this point in the history

  4. [tests] Fix typo in comment 

    The automated CI tests were using the correct index
    @jameshadfield
    jameshadfield committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    ecf2134 View commit details
    Browse the repository at this point in the history

  5. [version descriptors] redirect versioned URLs 

    The desired behaviour is that (e.g.) dengue/denv1@version redirects to
dengue/denv1/genome@version, similarly to how the non-versioned URL
currently behaves. This failed because of three inter-related aspects:

1. The previous commit removed (e.g.) "dengue/denv1" from the index,
   choosing instead to place that data under the "dengue/denv1/genome"
   ID.
2. The server instantiates the Resource (Dataset) class for the request
   before resolving/redirecting the URL. During this instantiation the
   requested version was checked against the index and since the
   resource wasn't present in the index this failed (NotFound error).
   This was solved by deferring the version lookup until after the URL
   resolving/redirecting stage.
3. The redirect URL didn't include the version descriptor.
    @jameshadfield
    jameshadfield committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    c88a87d View commit details
    Browse the repository at this point in the history