Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: use etags for caching to avoid backend/db load when idle #2731

Merged
merged 34 commits into from
Sep 17, 2024
Merged

feat: use etags for caching to avoid backend/db load when idle #2731

merged 34 commits into from
Sep 17, 2024

Commits on Sep 12, 2024

  1. feat: add table_update_tracker tracking last change per table 

    co-authored with @anna-parker and Claude Sonnet
    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    482159b View commit details
    Browse the repository at this point in the history

  2. try

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ae5753f View commit details
    Browse the repository at this point in the history

  3. now it might work?

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ac7a7f5 View commit details
    Browse the repository at this point in the history

  4. Remove views

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4d3816f View commit details
    Browse the repository at this point in the history

  5. fixup

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b244c33 View commit details
    Browse the repository at this point in the history

  6. Change to a conditional update for useNewerProcessingPipelineIfPossib… 

    …le, add last_updated_db as a view. (#2741)

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>
    @corneliusroemer @anna-parker
    corneliusroemer and anna-parker committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    2dfbc02 View commit details
    Browse the repository at this point in the history

  7. Still return current newest version

    @anna-parker @corneliusroemer
    anna-parker authored and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8c89e32 View commit details
    Browse the repository at this point in the history

  8. Return version from correct place

    @anna-parker @corneliusroemer
    anna-parker authored and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b086bc5 View commit details
    Browse the repository at this point in the history

  9. Don't issue delete statement unless there's something to delete

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    811b129 View commit details
    Browse the repository at this point in the history

  10. fix ktlint

    @corneliusroemer
    corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    a1f37ad View commit details
    Browse the repository at this point in the history

  11. Adjust useNewerProcessingPipelineIfPossible to only update table when… 

    … there are changes
    @anna-parker @corneliusroemer
    anna-parker authored and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    2931c55 View commit details
    Browse the repository at this point in the history

  12. Table update tracker anya (#2764) 

    * Add option to get-released-data to check last update time.

* make name better

* Make or short-circuiting

* Only work with table_update_tracker - return lastTime in LAST_MODIFIED header

* Remove unused functions

* add back missing imports - somehow format removed this???

* Update bash scripts

* Attempt to fix bash

* 2nd little fix

* echo value of last_modified not name

* Do not update SILO if loculus responds with 304.

* Always echo $lastSnapshot unless new data existed in loculus AND this was successfully sent to LAPIS - then echo $last_modified

* save last snapshot to file

* Update with loculus data anyways if longer than 1hour since last update

* fix

* Improve silo import scripts etc

* add back BACKEND_BASE_URL

* Log headers

* Regularize

* Better logs

* Try to fix error

* return 0 and never null in header

* Fix read non-existent file bug

* Better wrapper logging

* fix delete after update snapshot

* Make grep case insensitive

* add back removed exit 0

* Improvement to logging

* Add test

* Add full process to test

* Update documentation

---------

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
    @anna-parker @corneliusroemer
    anna-parker and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    453b826 View commit details
    Browse the repository at this point in the history

  13. CR edits to table-update-tracker (#2767) 

    * Add option to get-released-data to check last update time.

* make name better

* Make or short-circuiting

* Only work with table_update_tracker - return lastTime in LAST_MODIFIED header

* Remove unused functions

* add back missing imports - somehow format removed this???

* Update bash scripts

* Attempt to fix bash

* 2nd little fix

* echo value of last_modified not name

* Do not update SILO if loculus responds with 304.

* Always echo $lastSnapshot unless new data existed in loculus AND this was successfully sent to LAPIS - then echo $last_modified

* save last snapshot to file

* Update with loculus data anyways if longer than 1hour since last update

* fix

* Improve silo import scripts etc

* add back BACKEND_BASE_URL

* Log headers

* Regularize

* Better logs

* Try to fix error

* return 0 and never null in header

* Fix read non-existent file bug

* Better wrapper logging

* fix delete after update snapshot

* Make grep case insensitive

* add back removed exit 0

* Improvement to logging

* Add test

* Add full process to test

* Update documentation

* Use etag instead of modified-since for spec compliance

* ktlin fix

* Move logic into model

* Fix etag formatting

* Fix etag yet again

---------

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>
    @corneliusroemer @anna-parker
    corneliusroemer and anna-parker committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4cefe82 View commit details
    Browse the repository at this point in the history

  14. refactor(backend): streamline control flow

    @fengelniederhammer @corneliusroemer
    fengelniederhammer authored and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    610751d View commit details
    Browse the repository at this point in the history

  15. refactor(backend): extract function

    @fengelniederhammer @corneliusroemer
    fengelniederhammer authored and corneliusroemer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4b723e4 View commit details
    Browse the repository at this point in the history

  16. Update backend/src/main/kotlin/org/loculus/backend/controller/Submiss… 

    …ionController.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    08dd1dd View commit details
    Browse the repository at this point in the history

  17. Update backend/src/main/kotlin/org/loculus/backend/controller/Submiss… 

    …ionController.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    f3ae9a7 View commit details
    Browse the repository at this point in the history

  18. Update backend/src/test/kotlin/org/loculus/backend/controller/submiss… 

    …ion/GetReleasedDataEndpointTest.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ef6ca41 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Merge branch 'main' into table-update-tracker

    @corneliusroemer
    corneliusroemer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ec66ac7 View commit details
    Browse the repository at this point in the history

  2. Format

    @corneliusroemer
    corneliusroemer committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    42342c3 View commit details
    Browse the repository at this point in the history

  3. Update backend/src/main/kotlin/org/loculus/backend/service/submission… 

    …/UpdateTrackerTable.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    697ddeb View commit details
    Browse the repository at this point in the history

  4. Update backend/src/main/kotlin/org/loculus/backend/service/submission… 

    …/UpdateTrackerTable.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    a96bf51 View commit details
    Browse the repository at this point in the history

  5. Update backend/src/main/resources/db/migration/V1.2__add_table_update… 

    …_tracker.sql

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    9b0ff25 View commit details
    Browse the repository at this point in the history

  6. Update backend/src/main/kotlin/org/loculus/backend/model/ReleasedData… 

    …Model.kt
    @corneliusroemer
    corneliusroemer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    840208e View commit details
    Browse the repository at this point in the history

  7. Update backend/src/main/kotlin/org/loculus/backend/service/submission… 

    …/SubmissionDatabaseService.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    52f1421 View commit details
    Browse the repository at this point in the history

  8. Automated backend code formatting

    Loculus bot committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    eb5a430 View commit details
    Browse the repository at this point in the history

  9. Update backend/src/test/kotlin/org/loculus/backend/controller/submiss… 

    …ion/GetReleasedDataEndpointTest.kt

Co-authored-by: Fabian Engelniederhammer <92720311+fengelniederhammer@users.noreply.github.com>
    @corneliusroemer @fengelniederhammer
    corneliusroemer and fengelniederhammer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    2ec712e View commit details
    Browse the repository at this point in the history

  10. Add import, format

    @corneliusroemer
    corneliusroemer committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    27760ba View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into table-update-tracker

    @corneliusroemer
    corneliusroemer authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    8f3d044 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Formating updates

    @anna-parker
    anna-parker committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9268b63 View commit details
    Browse the repository at this point in the history

  2. Update tests

    @anna-parker
    anna-parker committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    77a533b View commit details
    Browse the repository at this point in the history

  3. Allow tableName selection in getLastDatabaseWriteETag.

    @anna-parker
    anna-parker committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    117775e View commit details
    Browse the repository at this point in the history

  4. Make list of tables used by get-released-data etag explicit: RELEASED… 

    …_DATA_RELATED_TABLES
    @anna-parker
    anna-parker committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7581f22 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into table-update-tracker

    @corneliusroemer
    corneliusroemer authored Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ead8bb3 View commit details
    Browse the repository at this point in the history