Best approach for Pager and pagination

[electrolobzik]

In my app, I had to implement a state machine for pagination, which uses MutableStore and Pager under the hood. From my point of view Pager and this state machine have an intersection in responsibility, so maybe the best approach would be to implement a single stateful component replacing the current Pager and giving the functionality of error handling and state management.

[matt-ramotar]

Thanks for starting this discussion. Can you take a look at these changes to RealPager and let me know if it's what you have in mind? https://github.com/MobileNativeFoundation/Store/pull/606/files#diff-8ed40e56a1ae3e29a62909ddb06a0b8d582a768df63dbee7636020ac7cf56bfe

[electrolobzik]

The main concern regarding using Pager just as a source of and as flows is the point that in real applications the pagination logic should be more complex. For example, it should be able to handle errors in the context of the operation where this error occurred and should distinguish between errors in loading from the cache, and from remote for the initial loading and for the next page loading. Also, it should handle cases like "the next page is empty" to avoid extra requests on scrolling or even show a different UI for this case. This means that this logic should be anyway present somewhere. I am not sure if the Store library is the correct way to place this logic. But I see some parts of it in the RealPager. For example, it is already merging data after loading the next page and serving that data as a whole piece. If there is another stateful component that will handle all that logic that I mentioned above it will in the best case duplicate some parts of the Pager and in the worst case will require a different implementation of the Pager to have access to the raw data that came from the Store.

[matt-ramotar]

I'm not sure I'm following - Can you please clarify what functionality is missing and requires code duplication? If you could share sample code that would be helpful

[electrolobzik]

I already see that the Pager is serving the commutative data, but I want to distinguish the result of loading and move the state machine in a special state when the next page is empty. I need access to the raw data which is not possible with the current implementation of the Pager. So the Pager is currently doing the following things (maybe it is not the full list):

1. It loads data
2. It handles the decomposition of the data to correctly cache it
3. It merges pages and provides the commutative data to the user

In my case, the p3 duplicates the corresponding logic in the Paginator, which I shared, and also doesn't allow access to raw data.
In other words, I would like to see the component which will either do all in one (not just loading and merging but also will have state machine, so I will not need that logic outside) or have only one responsibility (p2).
Probably it could be both: one separate component (decomposition adapter) for loading collections from the Store with transparent caching and another one, separate which manages the state of the particular use-case (screen) and using the adapter under the hood. In my case I am trying to use the RealPager as such adapter (but it doesn't fit well now) and have Paginator as a high-level manager for everything.

Let me also share an example of a repository with Paginator: https://gist.github.com/electrolobzik/a52637aae091acc23d88a1bdf6a4e885
It is also just a draft and I don't like the level of complexity here. I suppose it comes from the fact that there are too many layers using it's own data classes in API. If it would be possible to unify all this it will simplify the repository. Having the Paginator (or some equivalent), using decomposition-adapter inside in the library itself would simplify it even more and will allow to get rid of the Pager in the repository.

[electrolobzik]

So this is how I see it:
( Pager ( Decomposition-Adapter ( Store ) ) )

1. If one needs just Store with plain data - he can take just Store
2. If one needs to implement a general screen with pagination - he takes the Pager and the pager manages everything via provided store and some rules
3. If one needs some even more complex logic, where for some reason he needs to load collections, but it is not a regular pagination task, or he just needs to load a bunch of data without further pagination and have it cached properly - he takes Decomposition-Adapter with a Store.

[electrolobzik]

I've shared an example of the state machine I am talking about in this gist: https://gist.github.com/electrolobzik/8976a32c31b79ff638e4d7ad6c10eb34. Please do not judge strictly, it is a draft and has not been tested yet. But it should give the idea. The state of the Paginator is later mapped to the UI state and rendered (in my case reactively with Compose).

[matt-ramotar]

Thanks for sharing. It's not clear to me what requires code duplication - Pager provides a stream of data and errors, which could be mapped and provided to your state machine.

Re proposal to create a single stateful component - I'd be concerned about combining this with Pager. I think it would be preferable to keep them separate but allow them to interact through a clear interface. Separately, not all users will need this. And the users that do, I expect they will want implementation deferred to them, because it's going to be so custom

[electrolobzik]

The tree of discussion messed up a bit :).

1. Above I have shared the proposal for the separation of responsibility between components. I would separate the decomposition-adapter the only responsibility of which would be transparent caching of collections inside the store. This component can be used inside a more complex stateful component which will be used for pagination or this adapter can be used separately for loading collections. It could be even a component of a Store like fetcher, sourceOfTruth etc. It needs to be figured out what is the best architecture here. But the main point is that it is separate from the Pager itself.

2. Regarding the point that the implementation of the "pagination manager" should be deferred to the end library user. This may be the case and if the "adapter" from p1 will be separate it will be easy to make any custom implementation of page loading because the library will provide everything required including the mechanism to load and cache both collections and single items. But I suppose there could be the most common implementation which will cover 80% оf use cases and it would be helpful to have one in the library. Another thought here is that the current Pager implementation now already tries to have a simple logic of page loading and has state, and it probably makes sense to either improve it to cover more cases or remove that responsibility from the library totally.

[electrolobzik]

There are two more considerations why there should be 2 separate and independent components for pagination and collection decomposition:

1. In some cases caching is not required and the backend API should be directly connected to the pagination mechanism without any other processing. That means that the Pager should not rely on the current Store API and should take some interface like this one as a source of data:

interface PaginationDataProvider<Data : Any, Cursor : Any> {
    suspend fun loadInitialData(): Flow<PaginationDataResult<Data, Cursor>>
    suspend fun loadPage(key: Cursor): Flow<PaginationDataResult<Data, Cursor>>
}

typealias PaginationDataResult<Data, Cursor> = DataResult<DataWithCursor<Data, Cursor>>

data class DataWithCursor<Data : Any, Cursor : Any>(
    val cursorForLastPage: Cursor,
    val items: List<Data>
)

2. In some other cases we can just want to pull a collection of elements and cache it transparently. No pagination is required in this case. So the Store (or some additional API) should provide this opportunity.

I tried to illustrate the idea in the following diagram showing the possible use cases and required blocks:

[electrolobzik]

I would propose to add "Full" as another possible action of the PagerState. That state means that the previous loading more was successful, but returned empty page, meaning that there is no more data to load. The Pager should not try to load more from this state. Also I would propose adding a separate state "InitialLoadingError" like the existing "ErrorLoadingMore".

[matt-ramotar]

• Data has a nullable next page. My thought was that would be sufficient for representing there is no more data to load.
• There is a separate Error state. This will be used only when we don't already have data (e.g., initial load)

Let me know if the above works for you

[matt-ramotar]

@electrolobzik Added a README

[matt-ramotar]

@electrolobzik - Let's move further discussion to #611. Please feel free to leave comments and questions there