KHC Site Crawler is a fast, multi-threaded web crawler built with Python and Flask. It searches keywords across websites without native search, providing deep, concurrent crawling with real-time results, proxy support, and export features. Ideal for developers needing efficient keyword discovery on any domain.
Generate clean, AI-ready llms.txt files for your website or docs. Supports crawling, sitemaps, static builds, and framework-aware adapters (Next.js, Vite, Nuxt, Astro, Remix). Includes Markdown/MDX docs mode and robots.txt generator for LLM and search crawlers.
Site crawler used in Ethora platform as an option to import your specific business data into your AI agent chat bot.
Add a description, image, and links to the site-crawler topic page so that developers can more easily learn about it.
To associate your repository with the site-crawler topic, visit your repo's landing page and select "manage topics."