Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chunked package lists #1035

Merged
merged 6 commits into from
Sep 19, 2024
Merged

Chunked package lists #1035

merged 6 commits into from
Sep 19, 2024

Commits on Sep 3, 2024

  1. Add APIV1ChunkedPackageCache 

    PackageListing list for Lethal Company has grown so large, that the
download timeouts for people with slow connections. Furthermore sooner
or later we will go over the limit of data we can pass to mod manager's
plugin layer in one go.

To solve these issues, the package list will be downloaded and
processed in chunks. API will provide an index file which contains URLs
to the chunks of package info. All data is served in gzip compressed
blob files so CDN can cache it efficiently.
    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    ca2abd6 View commit details
    Browse the repository at this point in the history

  2. Add periodic task for updating APIV1ChunkedPackageCache 

    New cache files are created at the top of the hour and the stale cache
is dropped. Note that currently the latter is not implemented, since we
have to figure out how to safely delete blob files from main and mirror
storages without ending up in situations where the file is deleted from
one but still exists in the other due to some error.
    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    0d6622a View commit details
    Browse the repository at this point in the history

  3. Add API endpoint for fetching APIV1ChunkedPackageCache index blob

    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    409abc8 View commit details
    Browse the repository at this point in the history

  4. Fix multiple issues in APIV1ChunkedPackageCache 

    - Use ForeignKey instead of OneToOneField for index, since the same
  blob will be shared between different versions of the cache if the
  contents haven't changed
- Add CACHE_CUTOFF_HOURS and UNCOMPRESSED_CHUNK_LIMIT as class
  constants and use the latter as a default parameter for
  update_for_community(). This makes it easier to write unit tests
- Fix update_for_community() to not create empty first chunk if the
  data from a single PackageListing would exceed the chunk size limit
- Fix update_for_community() to create a chunk (containing an empty
  list) if the community has no packages at all
- Fix get_latest_for_community() to return None instead of throwing if
  community has no cache
- Use fixed mtime parameter when calling gzip.compress() to ensure that
  same input bytes result in same output bytes, which is important
  since the hash of the bytes is used as the cache key
- Add get_blob_content() helper to avoid repetition in unit tests
    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    ecb2c4f View commit details
    Browse the repository at this point in the history

  5. Add unit tests for APIV1ChunkedPackageCache

    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    dc985ea View commit details
    Browse the repository at this point in the history

  6. Optimise APIV1ChunkedPackageCache memory usage 

    Reduce the amount of data Django's ORM keeps in memory at the same time
by processing the PackageListings in smaller, consecutive QuerySets.
    @anttimaki
    anttimaki committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    01b309d View commit details
    Browse the repository at this point in the history