Feature: Deep Blacklisting

[nautbot]

Develop deep blacklisting job/script to consume and process XML/JSON feed created in #144 per undetermined technique

Reference issue #20 for original /u/nautbot functionality

[nautbot]

Suggested solution per @jpleger:

it might actually be something that can be done in scrapy and splash
which isn't too much effort to setup either
https://scrapy.org/
https://github.com/scrapy-plugins/scrapy-splash
I haven't used either projects in the last couple years, but scrapy was pretty easy to work with and with splash, it adds js support
https://doc.scrapy.org/en/latest/topics/link-extractors.html#link-extractors
can use the link extractors to find all links on a website, and then write a simple middleware that logs all redirects after the frontier crawl
(first page that is)

[psineur]

This needs re-triage to v1 v1.1 MVP+ or v3.
v2 was tech-only stack/platform agnostic