This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
GoSitemap2Md is a Golang program that generates a sitemap URL in Markdown format and stores the URLs in a urls.json file for easy adding of new URLs. This tool simplifies the process of generating and maintaining a sitemap for your website.
Collect links through the sitemap.xml or robots.txt
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
a python script that crawls website sitemap in a very quick way with multi threading and extract, write SEO based data to CSV file
Add a description, image, and links to the sitemap-crawler topic page so that developers can more easily learn about it.
To associate your repository with the sitemap-crawler topic, visit your repo's landing page and select "manage topics."