NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Oct 28, 2024 - JavaScript
#
robots-exclusion-standard
Here are 10 public repositories matching this topic...
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
search-engine whitelist user-agent seo crawling twitterbot robots-txt googlebot crawlers web-crawling bingbot robots-exclusion-standard blocking-bots web-robots search-engine-optimization baiduspider
-
Updated
Feb 18, 2024
Alternative robots parser module for Python
-
Updated
Dec 9, 2024 - Python
Go robots.txt parser
-
Updated
Nov 27, 2017 - Go
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
Jun 23, 2023 - JavaScript
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
web-scraping robots-txt web-crawling hacktoberfest robots-exclusion-standard robots-exclusion-protocol robots-txt-parser
-
Updated
Nov 6, 2024 - PHP
A parser for robots.txt with support for wildcards. See also RFC 9309.
parser erlang parsing crawling robots-txt parsing-library robots-parser erlang-library robots-exclusion-standard rfc-9309
-
Updated
Nov 21, 2023 - Erlang
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
-
Updated
Aug 20, 2020 - C++
Fully native robots.txt parsing component without any dependencies.
-
Updated
Oct 8, 2022 - JavaScript
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag
-
Updated
May 2, 2024 - C#
Improve this page
Add a description, image, and links to the robots-exclusion-standard topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-exclusion-standard topic, visit your repo's landing page and select "manage topics."