About the project | Technologies | Getting Started | License
The Upwork Scanner is an application that was conceived as a stage in the selection process for a Scanner Enginner at Argyle.
I started the project trying to found a way to get informations without need a simulate a user in a browser.
My first try was open the site manually and inspect the requests that are been made, see the server endpoints, tokens, cookies and try simulate then with curl in my terminal, but it's fail. The process of login not is a simple request do a endpoint that return a jwt token, it's more complicated.
My second try was intercept the requests that are been made by site using a Burpsuite, that is a very common tool for security researches, but not was possible too.
So the only away that i founded was simulate a user using a browser with Selenium, that is a very used browser automation tool for python. I started the challenge with a just file using selenium and navigating in the page and getting informations. Also tried to intercept the request that are been made beetwen selenium browser and upwork server with Selenium Wire, but upwork blocked my automation when i did that, so i gave up on that idea.
After simple testing the flow with selenium, i refatored the project to follow the Clean Archicture, that is my favorite code design, and improve some aspects of the code.
The result of my last run is in jobs and user_informations.json, i used bobbybackupy user.
My principal difficulty in this challenge was work with python ecossystem, i usually work with Node, things like the dependencies management , docker configuration, lint checks, tests, things that could be better if i got used to the standards.
The project has improvent, like:
- Config dockerfile
- CI/CD with lint check
- Unit tests
- More resilient flow
- More login options
Technologies that was used to build this application
The dockerfile image not it's working, so it's necessary have installed the python and the poetry.
You need to add a .env file follow a .env.example.
Now it's just enter in the project and run:
poetry install
poetry run python src/main.pyand see the result in jobs.json and user_informations.json files that will be created
This project is licensed under the MIT License - see the LICENSE file for details.
Made with 💜 by Isaque Diniz 👋 See my linkedin