robots-txt-parser

Here are 5 public repositories matching this topic...

PHP class for parse all directives from robots.txt files according to specifications

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

Add a description, image, and links to the robots-txt-parser topic page so that developers can more easily learn about it.

To associate your repository with the robots-txt-parser topic, visit your repo's landing page and select "manage topics."