From e2e43cfa393e201afb4ea11082f68df0323b3c36 Mon Sep 17 00:00:00 2001
From: mavrick-1 <mavrickrishi@gmail.com>
Date: Thu, 27 Nov 2025 21:24:07 +0530
Subject: [PATCH] feat: add robots.txt to prevent crawler traps

This commit adds a robots.txt file to the public directory of the Next.js application.
The robots.txt file disallows crawling of search pages with query parameters to prevent web crawlers from getting stuck in crawler traps.

Fixes #335
---
 app/public/robots.txt | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 app/public/robots.txt

diff --git a/app/public/robots.txt b/app/public/robots.txt
new file mode 100644
index 00000000..2b2410e9
--- /dev/null
+++ b/app/public/robots.txt
@@ -0,0 +1,9 @@
+User-agent: *
+Disallow: /datasets?*
+Disallow: /tasks?*
+Disallow: /flows?*
+Disallow: /runs?*
+Allow: /datasets$
+Allow: /tasks$
+Allow: /flows$
+Allow: /runs$