From ca4ab789b127e0e83ae60429197ff3ef215fbbec Mon Sep 17 00:00:00 2001 From: Wade Barnes Date: Thu, 2 Nov 2023 08:59:59 -0700 Subject: [PATCH] Update robots.txt file. - Separate blocks are based on information here; https://www.feitsui.com/en/article/32 Signed-off-by: Wade Barnes --- public/robots.txt | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/public/robots.txt b/public/robots.txt index eb05362..ac53099 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,2 +1,18 @@ User-agent: * -Disallow: +Disallow: /api/ +Disallow: /hooks/ + +User-agent: Baiduspider +User-agent: 360Spider +User-agent: Yisouspider +User-agent: PetalBot +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: Sogou web spider +Disallow: / + +User-agent: Sogou inst spider +Disallow: / \ No newline at end of file