MiniCrawler

This script explores recursively all internal links of a given domain and retrives useful info (assets' url, inbound/outbound links between pages, etc...) for each of them.

IDEA

Script logic

A picture that I drew to visualize the problem before I started to code.

Usage

$ iex -S mix
iex(1)> MiniCrawler.start

Tested cases

• https://elixir-lang.org - depth: 6
• https://sedna.com - depth 4
• http://sinatrarb.com - depth 5 (best example)
• https://www.emojicode.org