Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add pre_navigation_hooks to PlaywrightCrawler #631

Merged
merged 1 commit into from
Oct 30, 2024

Conversation

Prathamesh010
Copy link
Contributor

@Prathamesh010 Prathamesh010 commented Oct 29, 2024

Description

Add a new decorator for processing pre navigation hooks

Example Use:

from crawlee.playwright_crawler import PlaywrightCrawler
from .routes import router

async def main() -> None:
    """The crawler entry point."""
    crawler = PlaywrightCrawler(
        request_handler=router,
        max_requests_per_crawl=50,
    )

    @crawler.pre_navigation_hook
    async def hooky(context) -> None:
        print(f'Hook1')

    @crawler.pre_navigation_hook
    async def hooky2(context) -> None:
        print(f'Hook2')

    await crawler.run(
        [
            'https://crawlee.dev',
        ]
    )

Issues

Checklist

  • CI passed

Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! Could you please add a test of the functionality as well? Also, some documentation would be great, if you have the time.

src/crawlee/playwright_crawler/_playwright_crawler.py Outdated Show resolved Hide resolved
@Prathamesh010
Copy link
Contributor Author

I have added a test and made the suggested changes. @janbuchar

Also, some documentation would be great, if you have the time.

What type of documentation needs to be added? Will it be an example or a guide or something else?

@janbuchar
Copy link
Collaborator

Just extending the PlaywrightCrawler example should be enough for now.

@Prathamesh010
Copy link
Contributor Author

Documentation is updated. @janbuchar

@B4nan B4nan changed the title feat: Add prenavigation hooks to playwrightCrawler feat: Add preNavigationHooks to PlaywrightCrawler Oct 29, 2024
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

docs/examples/code/playwright_crawler.py Outdated Show resolved Hide resolved
src/crawlee/playwright_crawler/_browser_page_context.py Outdated Show resolved Hide resolved
docs/examples/code/playwright_crawler.py Outdated Show resolved Hide resolved
src/crawlee/playwright_crawler/_playwright_crawler.py Outdated Show resolved Hide resolved
@B4nan B4nan changed the title feat: Add preNavigationHooks to PlaywrightCrawler feat: Add pre_navigation_hooks to PlaywrightCrawler Oct 30, 2024
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@janbuchar janbuchar merged commit 5dd5b60 into apify:master Oct 30, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for preNavigationHooks in Playwright
3 participants