Go-React Web Crawler

A fullstack web crawler built with Go (Gin), React, and MySQL

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact

About The Project

This full-stack project uses GoLang for the backend and React TypeScript for the frontend.

The UI is designed with shadcn/ui and TailwindCSS to create a fully responsive application. Users can input any URL they want to crawl, and the app returns the HTML content of that website. Additionally, the crawled URL’s output can be viewed in a table format, and detailed data can be explored through graphical visualizations on a separate details page.

(back to top)

Built With

• 
• 
• 
• 
• 
• 

(back to top)

Getting Started

Installation

1. Clone the repo

   git clone https://github.com/kiwiscode/go-react-crawler

If you are going to run it with Docker in a development environment, follow the step below:

2. Build and run with Docker Compose

   docker-compose up --build

If you want to run it manually in a development environment, follow the steps below:

2. Install Frontend Dependencies and Start Frontend (React)

   cd frontend
   npm install
   npm run dev

3. Install Backend Dependencies and Start Backend (Go)

   cd backend
   go mod tidy
   go run main.go
   

(back to top)

Usage

The video below demonstrates how to use the app step-by-step. It’s designed to help you quickly understand the main features and workflow.

usage-video.mp4

(back to top)

Roadmap

• Add authentication
• Add URL process UI
• Add start/stop analysis toggle for URL analysis
• Make UI responsive
• Add paginated and sortable table
• Add global filter to the table
• Add column filters to the table
• Add checkbox selection for analyses in the table
• Enable deleting or re-running selected analyses via checkboxes
• Add row click redirect to URL detail page
• Display URL details with charts on the URL detail page
• Show crawl status per URL as queued, running, done, or error

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

This project is licensed under the MIT License. See the LICENSE file for details.

(back to top)

Contact

Aykut Kav - @kiwisc0de - ayktkav@gmail.com

Project Link: https://github.com/kiwiscode/go-react-crawler

(back to top)