Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add async support #72

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Add async support #72

wants to merge 8 commits into from

Conversation

encryptize
Copy link

In some cases, scraping speed is of significant importance, so I decided to create a PR with the addition of an async version of this project. What remained was to update the tests and add a mention of async support in the README.

Most important changes:

  • Optimization of the function responsible for randomizing the useragent
  • A change of the HTTP client from requests to httpx. This change is to use one common library for both sync and async versions.
  • Create a shared base class for wrappers.

@Giglium
Copy link
Owner

Giglium commented Feb 23, 2025

Hi @encryptize,
Thanks for the amazing work you have done! In the next few days, I will adjust the test and review the PR more thoroughly. Great catch on optimizing the randomizing user agent!

Is it not discouraged to re-create the client every time? Shouldn't we create it in the constructor and reuse it every time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants