crawlerd

crawlerd is a crawler daemon for small crawling jobs.

commands

• make grpc-gen - generate grpc code
• make compose-up - run all crawlerd services in docker
• make run-e2e-tests - run e2e tests inside docker
• make run-worker-integration-tests - run integration tests inside docker

TODO

crawlerd is still not production ready, below is the list needed for first production release:

• Rotating IP
• HTTP/Socks Proxy
• Fix bugs