It allows you to download a website from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.

A tutorial and code samples of web scraping with PHP

A Simple Script To Scrape DuckDuckGo Search Results Using Python And Selenium WebDriver.

This a project to demonstrate the use of standard python libraries like os, urllib, HTMLParser to create a minimalist webpage crawler that crawls webpages on a website to gather hyperlinks (URLs)

💫 Crawl urls from a webpage and provide a DomCrawler with Scraper Library

Crawls a website to generate insights

Crawls website and collect SEO relevant data

Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.

Recursive website crawler

The most advanced Lightshot (or prnt.sc) scraper ever!

Java website crawler - library for analyze and testing websites

sponge is a website crawler and links downloader command-line tool

The most advanced Imgur scraper ever!

Sitesweeper is a python package to help you automate your web scraping process, outputting pages to a file

Created a website-crawler in bash. Note, it's for a specific website and will not work unless you know the site.