README.md

web-scraper

This project consist of two parts: scrapper and listener. Spider walks through the web-page and sends payloads, that it found to the listener. Listener waiting for payloads and sends it to POLYSE database using SDK.

Installing

go get github.com/polyse/web-scrapper

Usage

1. Import package import ws "github.com/polyse/web-scrapper"
2. Install and start RabbitMQ.
3. Start polySE database on <example_host>:<example_port>
4. Run new spider like :

       cd cmd\daemon
       go build
       daemon.exe

5. Run new listener like :

       cd cmd\listener
       go build
       listener.exe

6. Send POST-message with auth Bearer token like :

       localhost:7171/start?url=http://go-colly.org

7. Enjoy results.

Credits

1. go-colly
2. surferua
3. rabbitmq
4. wire