How to retrieve more than 1000 distinct papers on a particular query?

[sadovsd]

Hello,

I'm trying to use the paper bulk endpoint, but I just can't get any more than 1000 distinct results. The documentation said you can get up to 10 million results so I'm confused what I may be doing wrong. Is it a problem with the way I used offset? I'd appreciate any help.

`def fetch_papers(query, total_papers=2000, num_papers_per_request=500, year_range="2023-", field_of_study="Computer Science", min_citation_count=5):
all_papers = []
start_index = 0

while len(all_papers) < total_papers:
    url = f"https://api.semanticscholar.org/graph/v1/paper/search/bulk?query={query}&offset={start_index}&limit={num_papers_per_request}&fields=paperId,authors,publicationDate,abstract,referenceCount,citationCount,influentialCitationCount,journal,publicationTypes,publicationVenue,title&year={year_range}&fieldsOfStudy={field_of_study}&minCitationCount={min_citation_count}"
    response = requests.get(url)

    if response.status_code == 200:
        data = response.json()
        papers = data.get("data", [])
        all_papers.extend(papers)

        if len(papers) < num_papers_per_request or len(all_papers) >= total_papers:
            break

        start_index += num_papers_per_request
    else:
        print("Failed to fetch papers:", response.status_code)
        print("Failed to fetch papers:", response.text)
        break

return all_papers[:total_papers]`

[danielnsilva]

Are you getting a specific error message? This would help in identifying the issue. I'm still working on this in the semanticscholar package, but i know that a token is needed for continued requests (see this example). However, since this is a s2api-related issue, you might find more help in the Slack community.