A mini 'project' that allows user to scrape a certain category in Kijiji, returns new values during the runtime whenever it can find a new value. Overlapping items are not outputted, allowing for data scrapping usages and data science.
Enter your parameters in the main, including the specific page you are scraping in Kijiji and the amount of pages you want it to monitor. Then run runfile.py.
In case you want to leave this overnight, please ensure your computer is not a potato. Or just run it on AWS.
Grasped the basics of request and web scraping.
Problems solved:
- Preventing log overlaps
- Switching pages
- Webhook breaking on duplicate runs
Beautiful soup for web scraping
Requests for accessing the web