This Node.js web crawler is designed to fetch and analyze links from a specified website. It focuses on exploring the links provided on the main page, recursively navigating through linked pages, and gathering information about link occurrences. The application is built with Node.js and includes a comprehensive test suite to ensure its reliability.
Clone the project
git clone https://github.com/bupd/webcrawler.git
Go to the project directory
cd webcrawler
Install dependencies
npm install
Start the crawler
npm run start <Website Link Here>
- Link Fetching: The application start fetching links from the main page of the provided website.
- Recursive Crawling: It recursively searches through all the links on the linked pages, focusing on the main domain and excluding external links and subdomains.
- Link Occurrences: The application counts and displays how many times a link is mentioned and referenced across the website.
- Command-Line Execution: Run the application by executing
npm run start <Website Link>
to initiate the crawling process.
To run tests, run the following command
npm run test
Contributions are welcomed! If you find issues or have suggestions for improvements, please open an issue or submit a pull request.
- Fork the repository.
- Create your feature branch: git checkout -b feature/new-feature.
- Commit your changes: git commit -m 'Add new feature'.
- Push to the branch: git push origin feature/new-feature.
- Open a pull request.
For support, email bupdprasanth@gmail.com or join our Discord channel.
- The application leverages Node.js for its functionality. Happy crawling!