Merge pull request #202 from WadeBarnes/update-robots-txt

Update robots.txt file.
bcgov · Nov 2, 2023 · 8d3ca16 · 8d3ca16
2 parents 448de4b + ca4ab78
commit 8d3ca16
Showing 1 changed file with 17 additions and 1 deletion.
diff --git a/public/robots.txt b/public/robots.txt
@@ -1,2 +1,18 @@
 User-agent: *
-Disallow:
+Disallow: /api/
+Disallow: /hooks/
+
+User-agent: Baiduspider
+User-agent: 360Spider
+User-agent: Yisouspider
+User-agent: PetalBot
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: Sogou web spider
+Disallow: /
+
+User-agent: Sogou inst spider
+Disallow: /