Update robots.txt file.

- Separate blocks are based on information here; https://www.feitsui.com/en/article/32

Signed-off-by: Wade Barnes <wade@neoterictech.ca>

public/robots.txt

@@ -1,2 +1,18 @@
 User-agent: *
-Disallow:
+Disallow: /api/
+Disallow: /hooks/
+
+User-agent: Baiduspider
+User-agent: 360Spider
+User-agent: Yisouspider
+User-agent: PetalBot
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: Sogou web spider
+Disallow: /
+
+User-agent: Sogou inst spider
+Disallow: /