Library Scanning problems

[SpaceFrags]

Hello!

I am selecting my games folder on my Linux server, but it is subdivided in different folders:

• Steam: Official steam folder where install own steam games, which also have other steam files created by Steam.
• ROMs: Roms files for emulators with subfolder (ps2, psx, nes, snes, n64, gc, wii, etc.).
• Others: For non-steam games that I added to my library of Steam.

This obviously creates some problems during the scans as it finds a lot of files that thinks are games when they are not, mostly two main things:

• RetroArch standalone or Steam version: This "game" have lots of exectutabable files, like bios files and others that are picked up, but at the same time the game itself is not picked up as the linux executable does not really have an extension as it is an AppRun from an extrated .appimage file.
• Steam: The steam folder contains the official files of steam when add a mount in the steam settings, as such it does not have only the "common" folder that contains the games, but also some other files that get picked up.

I dunno if it would be necessary to maybe add the possibility to have more folder to for the library scan so skipping on some problems, evnm if in this case games on steam like Retroarch would pick up a lot of false positive. I would assume a more integrated check from Playerr could help with a database of the files that are surely not game executable like Bioses or .exe files that are in some werid locations as shown in the picture down here with some example of things it picked up, for some reason with windows paths on linux.

Or, even having a blacklist button to specify which .exe files to skip as for example my version of reTHAWed has different exe files (reTHAWed, THAW, Updater) and all of them got in as separate games, even when it is just one!

At the same time introduce the possibility to have the AppRun as an accetable exectutable if there is an appimage close by or something similar.

Images

These are just examples, so there are many many mores. If u need it, I coluld send the entire paths that are picked up to put it maybe in a block list or something similar.

Also as you can see in the picture, the library scan is taking hours, I should be now into the 2 hours mark... most probably is checking too many small files or at some points it slow down as till the 80s games it was fast and then is going forward but taking so much time!

[Maikboarder]

Huge thanks for the details. To be honest, I am testing with my own personal library which is nowhere near as large or complex as yours, so I completely missed these performance bottlenecks and specific edge cases.

Based on your feedback, I’ve just refactored the scanner to:

Skip deep system folders (this should fix the multi-hour scan times).

Group files smartly (fixing the duplicates/junk issue).

Support AppRun/AppImages properly.

Please give the latest build a try. Since I can't replicate a library of your scale on my end, I’m relying on your test to confirm if we've finally nailed it. Fingers crossed!

[SpaceFrags]

I would love to try it, but I'm not seeing any new build? Is there a new beta build that I should change my Docker Compose with or something as I'm not seeing any update.

[Maikboarder]

I'm sorry, this weekend I was so busy!

I made a Update (v0.4.3) which specifically addresses your feedback:

Faster Scanning: I've replaced the old scanning method with a better system. It now skips junk folders and I hope it finishes in minutes instead of hours.
No More Duplicates: if a game has multiple .exe files (like your reTHAWed example), only the main one is added to your library.
Linux/Steam Optimization: I added explicit support for Linux AppRun files and expanded the global blacklist to ignore Steam system files and BIOS folders automatically (I hope It will works, I use to test on mac, my old windows and the only linux I test is a Raspberrypi)
Junk Filtering: the detection now strips version junk and avoids picking up secondary system tools as games.
Give v0.4.3 a try, it should make your library much cleaner! Thank you for your patience!

[SpaceFrags]

Ok, So I tried this new version and I would say it is Hit ot Miss.

From my setup that I said in the beginning 3/4 games from the Others folder were found, which are normal windows games with .exe files, instead the forth being RetroArch, being a "game", but more of a set of emulator I would assume it is not on IGDB and as such not found, but for some reason it searches and pick up random files from the config files of Retroarch, super deep down.

Completely opposite to this is the Steam folder, which didn't even try to go into it and didn't pick up any game, even if there is some games after getting slightly deep down in the common folder of Steam.

Finally, but understandable as for now there is not a emulator support, with this new version it took just one random rom for each rom folder, but compared to before, all the games that it picked up, not evenm one is correct, it match some similar, but random games, when before it was picking up all the games.

Let me show you a little tree of my games in general, it is a summary by showing some of the things that he found, not all.

games_list.txt

In general out of 71 games with the roms and Retroarch as a game:

• Version 0.4.1 - Found 94 games where in those 70/71 real games and perfectly matched, but ok, IGDB do not consider Retroarch as game.
• Version 0.4.3 - Found 21 games, where in those 3/71 real games and perfectly matched, the others random files and some roms badly matched.

Without considering the roms, that I understand as for now are not supported, so it picks only one file in each folder and not all of them (even if the matching is random compared to before that it was perfect), I do not understand why it didn't pick up the Steam Games as shown in the file above.

For Retroarch, which is there, I would assume that if it was on IGDB maybe it would pick up the AppRun or .appimage as it states Retroarch in the name.

Maybe the possibility to give multiple game locations for scanning, as I have Steam installed games and non-Steam games I could give instead of the Other folder and the common folder of Steam so it would scan better, by leaving Retroarch and Roms out, with maybe in the future possibility, maybe, to have also roms folders, which in that case it takes not a single executable, but all from the console folder like before.

Or, a more intelligent version of scanning that understands if an executable is a text file, as in linux for example a file without extension could be a text file or a script, so not that easy to check like in Windows, need to open it and see kinda fully!

Maybe also giving possibility to blocking a specific game path to appear, so like a blacklistso it won't appear again in the next scan.

Thank you again, as you are making an amazing project, surelly need to understand how to do some stuff as Games are surely more complicated than just movies and Tv Shows ahahah!

Good Luck

[SpaceFrags]

I also want to attach some images of some games and path that are being associated like in the game list sent above:

[Maikboarder]

I did one step back. The lastest logic broke the searches. By the way I implemented the old logic so It can find results solwly but in the efficient way.

[SpaceFrags]

This version found more games between the others and ROMs, even some folders of roms found them all and some just one for folder and completely nothing was picked up in the RetroArch folder which is incredible.

What still wasn't matched was the Steam games, and some games in Roms.

But generally speaking, if these two could be matched, the sync could work really well.

This version still good, much faster in finding the games, even if a lot could not be found

[Maikboarder]

Hi!

Nice 5o hear it. Playerr still have no recognition of ROMs thats why you have some roms recognized. Now i'm focused on dbi connection for NS nsp xci archives. Maybe on next updates I can add festures with Roms.

On the other hand, you can sync your steam library via API if you use the windows/linux/mac app version and launch from the app.

Regards!

[SpaceFrags]

Indeed I could, but that would bring the entire library of games and not just those are installed on my Linux server, considering that I have two main places where I have steam with different installed games.

With all the games with the Steam API it would add more than 250 games, mostly trashy stuff, and it won't make me manage my installed games on my server, in that case would be simpler to just scan the game installed in my Steam folder like it scanned them at the beginning, but skipping maybe stuff that are not games, like all the Proton's versions and all the SteamLinuxRuntimes:

P.S.: Thx u for maybe implementing ROMs next!!! 🤩🤩

[Maikboarder]

OK!!! I will take your info and add it to the scan system. I need to go slowly with this kind of changes...Step by step. Today I will launch the usb DBI transfer protocol for the NS .nsp format! :)

[SpaceFrags]

Yes yes, take your time, no need to rush!

You are already making an amazing project and we are just loving it!

Whenever it is the time for it, i will be there for it! 💪💪