Search in CombinatorialLibrary.plot of combinatorial.py does not use pagination properly

[fabianschoeppach]

Hey guys, 👋

I've used the code in combinatorial.py as a template for my own plugin and, as is, the search in the CombinatorialLibrary class to find every CombinatorialSample referencing it should not work as intended.

For now it's (pseudocode incomming)

query = { ... }
search_result = search(query=query, pagination=MetadataPagination(page_size=1), ...)
for result in search_result.data:
    ...
    <extract data>

My understanding is that search_result.data is maximum page_size. Since it's probably unknown how many CombinatorialSamples are referencing a given Library, this should lead to issues.

I would propose something like this:

query = { ... }
page_after_value = None

while True:
    search_result = search(
        query=query,
        pagination=MetadataPagination(page_size=1, page_after_value=page_after_value),
        ...)

    # Process search results
    # actually this for loop is not needed if page_size ==1 
    for result in search_result.data: 
        ...
        <extract data> 
    
    # Check if there are more pages
    if search_result.pagination.next_page_after_value:
        page_after_value = search_result.pagination.next_page_after_value
    else:
        break

I've already talked about this with @hampusnasstrom. He didn't have this issue in the past, but maybe the underlying code has changed since then.

Maybe, as a first step, someone else could have a look, if they encounter this issue as well.