Epic games store scraper

This scraper is a small part of a larger project and is meant to scrap all the games from the epic games store.

One issue with the store is that it uses React to render the frontend, meaning we aren't able to make a requests to the different URL and process the DOM as no games will be available.

The work around is a bit meh but I am new to Rust and I am trying my best.

I created a small API with hyper that will process the DOM into a array of games using this format;

Array<{
  url: String,
  name: String,
  id: String,
  type: "product" | "bundles"
}>

What is required?

• Postman
• Rust

How to scrape?

1. Clone this repo and run cargo run, that will start a local server at http://127.0.0.1:3000/.
2. Open Postman and create a POST request with a endpoint of http://127.0.0.1:3000/scrape.
3. Open Epic Games, you are able to do this for individual categories or just all of them. Remember to set the pageSize to 1000 (as high as it can go).
4. Run the JS below and copy the output.
5. Back in Postman, paste the output in Body > Raw and make sure it is set to JSON.
6. Send the request.

JS Code

(function() {
  console.log(
    JSON.stringify({
      dom: document.getElementById("dieselReactWrapper").outerHTML
    })
  )
})()

That should create a file on your system called scrapped-text.json with the scrapped data.

At this point you are feel to do what you want with this data.

There are some weird edge cases that are missed. For example Ark, there are about 400 different editions/addons that each have their own page, but are only accessible through the Ark homepage and aren't in the listing page. I can imagine that it would be applicable to many games who have different editions/DLC.

What am I doing with it?

Coming soon...