Authentication issue: "detail": "Invalid page: Deep API pagination is not allowed. Please review API documentation."

[eshanprashar]

Hi Team,

First of all, congratulations on building such an amazing resource and interface 👏🏼 - I've been using CourtListener for a week, and I'm a fan!

On to the issue, I'm a registered user on the website (username: eshan23) and have been trying to fetch data using the positions API. While the request works well for the first 100 pages, starting page 101, this is what I get in the browser:

_GET /api/rest/v3/positions/?page=101_

HTTP 404 Not Found
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "detail": "Invalid page: Deep API pagination is not allowed. Please review API documentation."
} 

I looked at the documentation:

• Authentication docs mention the rate limit is 5000 per hour for authenticated users. Per my account usage, I am definitely not hitting this limit
• Past issue in the forum #1497 mentions error code 429 while I am getting 404
• I cannot view page 101 onwards either on the browser, or through my python script, which is as follows:

base_url = "https://www.courtlistener.com/api/rest/v3/positions/"
all_data = fetch_data(base_url, max_pages=100)

# Function to create a session with the API token
def make_session():
    s = requests.session()
    s.headers.update({"Authorization": f"Token {api_token}"})
    return s

# Function to fetch data from the API endpoint with pagination handling for a limited number of pages
def fetch_data(url, max_pages):
    session = make_session()
    all_data = []
    page_count = 0
    
    while url and page_count < max_pages:
        response = session.get(url)
        
        if response.status_code != 200:
            print(f"Error: {response.status_code}, {response.text}")
            break
        
        data = response.json()
        all_data.extend(data['results']) # append data from the current page to the list

        if 'next' in data and data['next'] is not None: # next is the key for the next page
            url = data['next']
            page_count += 1
        else:
            break
    
    # testing: save data to a json file
    with open('testing_data_fetch.json', 'w') as f:
        json.dump(all_data, f)
    
    return all_data

And then I extract relevant fields from all_data

How could I extract positions data from the rest of the pages?

[mlissner]

Hi @eshanprashar, great to meet you. Try using API v4. It's not launched yet, but a major objective of it was to fix this. Just switch from v3 to v4 in your requests and I bet you'll be good to go.

[eshanprashar]

I tested this by incrementing page numbers for the URL:
https://www.courtlistener.com/api/rest/v4/positions/?order_by=-id%2C-date_created%2C-date_modified%22&page=100

and got the same issue beyond page 100 as in v3, i.e.

{
    "detail": "Invalid page: Deep API pagination is not allowed. Please review API documentation."
}

Do the ordering fields need to configured differently in the URL?

[mlissner]

Gotcha, so instead of using page arguments, use the "next" fields from the returned data, and, for now, it doesn't look like you can order by more than one cursor field, so, for example, this request:

https://www.courtlistener.com/api/rest/v4/positions/?order_by=-id

Returns a next value of:

https://www.courtlistener.com/api/rest/v4/positions/?cursor=cD01MjYxNw%3D%3D&order_by=-id

And I think by following those "next" fields, you should be able to go as high as you need.

Hopefully this helps! It's certainly helping us draft documentation for this feature!

[eshanprashar]

Got it! This approach is well-suited for picking up the next key-value pair from the json object in code, but page numbers provided an excellent way to manually test results from random page numbers on the browser. It seems like to go to page 101, now I will have to click the next URL 100 times on the browser, is that correct?
Another potential limitation of not having page numbers on the browser is lack of clarity around total pages. With v3, for example, I knew that positions API had 2565 pages, which helped me think through code structure, testing etc.

[mlissner]

Yeah, that's the nature of this one, unfortunately. You get performance, but you lose those features.

[eshanprashar]

:/ okay!