v1.1.1

Fixed

The Parser now also trims hidden whitespace characters that aren't covered by PHP's trim() function by default. Such characters at the beginning of a line can cause parsing to fail, because it's important that user-agent and rule lines actually start with the corresponding keywords.

v1.1.0

Added

Method isExplicitlyNotAllowedFor() that ignores rules for wildcard user-agent (*) and checks if some path is explicitly not allowed for a certain user-agent.

v1.0.0

Changed

• Required PHP version is now 8.0.

Added

• It now also parses Sitemap: lines. You can get all referenced sitemaps via the sitemaps() method of the RobotsTxt class.

v0.1.2

Fixed

• Also allow usage of crwlr/url 1.0 as it's not a problem at all and the PHP version requirement of this package is still ^7.4|^8.0.

v0.1.1

Changed

• Upgraded crwlr/url package version constraint.

v0.1.0

[0.1.0] - 2021-10-07

Added

• Parsing robots.txt files
  • user-agent groups
  • allow and disallow rule patterns
• Check if a certain uri/path is allowed for a certain bot/user-agent according to the definitions in robots.txt
• Implemented as described in https://datatracker.ietf.org/doc/html/draft-koster-rep