Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the method of gathering data (WebScrapper -> REST Calls) #4

Open
gabuscuv opened this issue Nov 9, 2022 · 1 comment
Open

Comments

@gabuscuv
Copy link

gabuscuv commented Nov 9, 2022

HowLongToBeat now uses XHR calls, So, the game data is not already embedded in the page, So It brokes the library.

Using the browser debugger, I noticed some XHR calls to "https://howlongtobeat.com/api/search"
that request data contents:

{"searchType":"games","searchTerms":["Gravity","Rush","2"],"searchPage":1,"size":20,"searchOptions":{"games":{"userId":0,"platform":"","sortCategory":"popular","rangeCategory":"main","rangeTime":{"min":null,"max":null},"gameplay":{"perspective":"","flow":"","genre":""},"rangeYear":{"min":"","max":""},"modifier":""},"users":{"sortCategory":"postcount"},"filter":"","sort":0,"randomizer":0}}

and the answer body contents:

{"color":"blue","title":"","category":"games","count":2,"pageCurrent":1,"pageTotal":1,"pageSize":20,"data":[{"count":2,"game_id":30313,"game_name":"Gravity Rush 2","game_name_date":0,"game_alias":"Gravity Daze 2","game_type":"game","game_image":"Gravity-Rush-2-ms.jpg","comp_lvl_combine":0,"comp_lvl_sp":1,"comp_lvl_co":1,"comp_lvl_mp":1,"comp_lvl_spd":1,"comp_main":71005,"comp_plus":122272,"comp_100":192082,"comp_all":117265,"comp_main_count":114,"comp_plus_count":182,"comp_100_count":72,"comp_all_count":368,"invested_co":0,"invested_mp":0,"invested_co_count":0,"invested_mp_count":0,"count_comp":648,"count_speedrun":0,"count_backlog":1461,"count_review":298,"review_score":77,"count_playing":10,"count_retired":79,"profile_dev":"Project Siren","profile_popular":82,"profile_steam":0,"profile_platform":"PlayStation 4","release_world":2017},{"count":2,"game_id":44680,"game_name":"Gravity Rush 2: The Ark of Time - Raven's Choice","game_name_date":0,"game_alias":"","game_type":"dlc","game_image":"Gravity-Rush-2-ms.jpg","comp_lvl_combine":0,"comp_lvl_sp":1,"comp_lvl_co":0,"comp_lvl_mp":0,"comp_lvl_spd":1,"comp_main":7191,"comp_plus":7987,"comp_100":9483,"comp_all":7807,"comp_main_count":42,"comp_plus_count":19,"comp_100_count":23,"comp_all_count":84,"invested_co":0,"invested_mp":0,"invested_co_count":0,"invested_mp_count":0,"count_comp":133,"count_speedrun":0,"count_backlog":45,"count_review":67,"review_score":73,"count_playing":0,"count_retired":2,"profile_dev":"Project Siren","profile_popular":9,"profile_steam":0,"profile_platform":"PlayStation 4","release_world":2017}],"displayModifier":null}

This change should be future-proof. (because using a Web Scraper could be an issue if the page changes too much the appearance or the visual representation) and easy to maintain.
I will try to implement it myself.

@gabuscuv
Copy link
Author

gabuscuv commented May 6, 2023

PR is Ready:
#7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant