recsource.tv data import

[ville-j]

I went through the trouble finding the recsource.tv data backups and now that replay uploading, comments and ratings are implemented on the new site it would be possible write scripts to import all that data. Is it something that's still wanted? Some people have said earlier that they would want it since they had uploaded nice catalog of replays to save etc. There are 46833 replays backuped, many of them are battle winning replays I suppose.

[pawq]

Absolutely yes!!! There are tons of recs that were lost with resource.tv, so recovering them would be amazing, especially with the "other replays in level" feature on elma.online. Hundreds of links also went rip, and what has been suggested before is to recover the old rec ids from rs and use them in new links, so that recsource.tv/usjgbha7wh could be replaced with elma.online/r/usjgbha7wh and it would work! Dunno if this is possible, but would be amazing and I'm sure greatly appreciated by a lot of people. Either way any effort to recover rs recs is mega appreciated!

[sunehs]

Yea I'd say still wanted. Many replays there that doesn't exist anywhere else, and getting them online would make it a nice resource for recs. Also hoping to import recs from other places, elmastats, kopasite etc.

For reference, the existing upload function:

elmaonline-site/src/utils/upload.js

Line 153 in 07e5b0c

export function uploadReplayS3(replayFile, folder, filename) {

can be imported with similar logic. For battle replays I suppose there's some indication that they're battle recs, I seem to remember they had a link to battle page at least, so we can skip those.

[ville-j]

Yeah there is some way (didn't check the code/db yet) to distinguish the battle replays from the other uploads so should be np.

[abulaatikko]

+1

(would be nice to see the best rated replays per year and the most viewed)

[ville-j]

The ratings in recsource are 1-5 and eolsite has 1-10 so if we want to import the ratings too then would be it ok to just multiply the recsource rating by 2? Comments can be imported and every upload can be assigned to right user if we can match the kuski names from recsource account to eol account which probably should be easy for most of the users.

[sunehs]

Yea that's probably best way to go about ratings. Could change the scale, but 1-10 is sort of set in stone for eol with chat votes, even if those are not directly related it's less confusing using the same. For comments, uploader id's can do a dry run and see if any known nicks have changed or something.

[sunehs]

@ville-j is this something you want to do? if not you can send me the necessary info to get the data and I can start looking at this

[ville-j]

I am/was going to do it but it's not priority, so you could do it too. I would need to create a user for you in aws etc, also fetching those replays will cost something so it should be handled carefully to avoid unnecessary expenses, but it's probably not that much anyway.

edit:
To clarify, the replay files itself are stored in s3 bucket and metadata is in a separate mysql db dump