NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Oct 28, 2024 - JavaScript
#
rfc-9309
Here are 3 public repositories matching this topic...
A parser for robots.txt with support for wildcards. See also RFC 9309.
parser erlang parsing crawling robots-txt parsing-library robots-parser erlang-library robots-exclusion-standard rfc-9309
-
Updated
Nov 21, 2023 - Erlang
Parse robots.txt and sitemaps using dotnet
-
Updated
Sep 2, 2023 - C#
Improve this page
Add a description, image, and links to the rfc-9309 topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the rfc-9309 topic, visit your repo's landing page and select "manage topics."