Skip to content

deploy-2025-01-28: Merge pull request #10397 from internetarchive/robots-txt-crawl-delays

Latest
Latest
Compare
Choose a tag to compare
@mekarpeles mekarpeles released this 28 Jan 21:31
· 63 commits to master since this release
deploy-2025-01-28
67903f4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode. 
Add crawl delays to robots.txt
Assets 2
Loading