How to crawl only the suggested site and not external sites #24
Unanswered
benzvi8888
asked this question in
Q&A
Replies: 1 comment
-
Unfortunately, I do not remember designing the code like this. The functionality is in these functions below, but when I designed this, modularity and best practice weren't in mind. Hence, I don't believe there will be an 'easy fix' unless you're willing to rewrite most of the Crawler class. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Seems when I checked the database the crawler go into social link and start index their content - it crawl Facebook, Twitter, LinkedIn and so on - it filled my database with it.
is there any way/option/code to limit the crawling to the site itself only?
Beta Was this translation helpful? Give feedback.
All reactions