Main generates min(n,8) new_user threads and main() waits for these threads to exit. Each new_user thread initiates a new connection to the server and retrieves a cookie. A shared queue q containing local files to crawl is passed to each thread. Each thread calls crawl(), which
- Retrieves filename by calling queue.get()
- creates a GET request to server for file
- downloads file flat to folder specified
- Finds all local links in that file if it is a text file and puts new links to queue.
- calls crawl() again with cookie
If queue is empty for 2 seconds crawl() returns and thread exits.
Delete download folder after use for correct behavior