Add support for robots.txt for any website

[ScottMansfield]

The robots.txt rules should survive restarts and be per-domain.

See http://www.robotstxt.org/robotstxt.html for some examples. I didn't find any standard java parsers onlien in a quick search, so maybe a custom parser is involved. It might be a separate project.

[ScottMansfield]

crawler-commons has a robots.txt parser:

https://github.com/crawler-commons/crawler-commons

[ScottMansfield]

This is being done in the terminator project

[ScottMansfield]

work done in 22b72fd and
5af8623