New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

canCrawl stalls in nodejs but not browser #16

Open

salivity opened this issue Jun 16, 2024 · 1 comment

salivity commented Jun 16, 2024

I created a project at https://davidclews.com/article/158.html and the robots.txt for https://www.homebase.co.uk/robots.txt appears a parse correctly but in node 20 when running canCrawl on a url the function stalls with no exception. What could this be?

Author

salivity commented Jun 16, 2024

did further checks seems to be this part

User-agent: *
Disallow: /*elysium.search***
Allow:  *******/*IFP=true*

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment