DM-47581: Add generic paginated query support #330
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rra
force-pushed
the
tickets/DM-47581
branch
from
0bc09e5 to
62ff262
Compare
rra
force-pushed
the
tickets/DM-47581
branch
3 times, most recently
from
ee5c331 to
703a279
Compare
fajpunk
reviewed
Nov 20, 2024
|
This is very very nice! |
Lift the pagninated query support from Gafaelfawr into Safir and make it generic so that it can be reused in Wobbly. Pagination support consists of a cursor class, which defines how results are ordered and paginated and how the cursor is serialized and deserialized, a paginated query runner that contains the logic for applying limits and keyset pagination, and a paginated result container that wraps a list of Pydantic models with metadata about the pagination. Also define a cursor type for the common case of results ordered by a `datetime` and then a unique key, in descending order, so that cursors for that case can be easily constructed. Fix the documentation build with current Sphinx. Adding `TypeVar` variables to the exports for documentation purposes is apparently now pointless since the documentation string appears to be forced to be generic documentation about type variables, so stop trying to include them.
A copy/paste error left the return type in the documentation set to a subclass instead of the parent class. Co-authored-by: Dan Fuchs <330402+fajpunk@users.noreply.github.com>
`PaginatedList` can hold any time of cursor, not just a `DatetimeIdCursor`. Co-authored-by: Dan Fuchs <330402+fajpunk@users.noreply.github.com>
Don't use the word limit, since that implies a SQL limit clause, which should not be applied outside of the pagination code. Co-authored-by: Dan Fuchs <330402+fajpunk@users.noreply.github.com>
Be explicit that the paginated query should not include a `LIMIT` directive. Co-authored-by: Dan Fuchs <330402+fajpunk@users.noreply.github.com>
Fix several minor documentation and comment issues caught during code review.
Return a next cursor if a previous cursor was used, even if the query had no limit. Properly construct previous cursors when following previous cursors backwards in the data. Randomize the test data and add more tests for the `Link` headers that would catch both of those problems.
rra
force-pushed
the
tickets/DM-47581
branch
from
6a97fba to
c487e58
Compare
Document that the `id_column` of a `DatetimeIdCursor` must be an integer. Document in multiple places that forward cursors include the entry they are based on but previous cursors do not and should return data starting with the entry immediately previous. Document that the query is run twice, once with restrictions and then again with `COUNT` to get the total number of available entries, and therefore the `COUNT` query needs to be fast (and thus based on indices or short tables).
Add separate methods to return the first, next, and previous URLs for a paginated list to make the HATEOS approach a bit easier, and implement `link_header` in terms of those methods.
The `query_object` and `query_row` methods had duplicate code for the unpaginated case. Refactor that into a helper method.
fajpunk
approved these changes
Nov 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Lift the pagninated query support from Gafaelfawr into Safir and make it generic so that it can be reused in Wobbly.
Pagination support consists of a cursor class, which defines how results are ordered and paginated and how the cursor is serialized and deserialized, a paginated query runner that contains the logic for applying limits and keyset pagination, and a paginated result container that wraps a list of Pydantic models with metadata about the pagination. Also define a cursor type for the common case of results ordered by a
datetimeand then a unique key, in descending order, so that cursors for that case can be easily constructed.Fix the documentation build with current Sphinx. Adding
TypeVarvariables to the exports for documentation purposes is apparently now pointless since the documentation string appears to be forced to be generic documentation about type variables, so stop trying to include them.