Is it possible to change the display of project.rs.xml in the open dialog to the name of the project?

[joewheaton]

@MattReimer please don't read or respond to this now... I am queuing up for when we finally sit down and prioritize warehouse development and make some decisions.

The Problem

For experienced users, we no longer think twice about the fact that every project (no matter how different) has the exact same name of project.rs.xml. However, this is just wrong. It is unintuitive to new and existing users.

The Request

Can we trick the display of the project name on open dialog to NOT be reading actual project.rs.xml file name on disk and instead read the <Name> as shown above? I'm suggesting this because I am guessing @MattReimer will not like idea of renaming project.rs.xml files in warehouse? I appreciate that you are likely using a bog-standard open file dialog/control, and this is not possible without making a custom interface for navigating disk to look for those.

The Acknowledgment

This has bugged me for a long time but I have never said anything about it (sorry). I like what we did in GCD (i.e. make a safe project name based on removing spaces and odd characters from equivalent of what's in and add our own custom extension (e.g. if Project <Name> was "Glenn Feshie" the file name would be GlennFeshie.gcd , even though *.gcd is really just an xml. That makes way more sense. The extension of that logic would be to have a file extension for every project type (e.g. for VBET maybe *.rs.vbet, for BRAT *.rs.brat, etc.). Maybe that is straightforward? Maybe that is a nightmare. However, I have had multiple users mention how this initially confused them.

[philipbaileynar]

Discussed with @joewheaton ....

Novice and less technically skilled users don't really get what is a project.rs.xml file. They know they have some data on disk organized as project, but they are unaware that each project has this special file with every project, much less that this is the file they should be selecting in a file browser dialog!

Solution

1. Do away with file browser dialogs!
2. We will ask the user to specify a top level "riverscapes data folder" on their system. This choice will get stored as a user preference (that they can change).
3. It is the users responsibility to download and unzip projects in any folder structure under this top level folder.
4. When users want to open a project in RAVE we will present them with a custom form that shows the folder hierarchy under their top level folder as a tree with the name of each project. We will build this tree by scraping their hard drive for project.rs.xml files. The custom project open form can have filters for specific project types etc.

This "curation of projects under a top level folder" is similar to how ArcCatalog works. You "connect" to a folder and then ESRI presents a curated view of datasets (not every single *.shp, *.dbf etc).

In summary, the goal is to present users with a more comprehensible view on the projects that they have on their computer. Users understand "I downloaded VBET for 17010203", they don't understand browsing to some obscure XML file that they didn't know existed.

[MattReimer]

I agree 100% that project.rs.xml files aren't intuitive but "Do away with file browser dialogs!" is not the way to go. In QGIS you can use the catalog, you can open a shp file manually OR you can drag a file on top of the app. Seems to me if we want our own project catalog feature we should implement it on top of what we already have and put a catalog icon into our toolbar.

I'm not the end user here, granted, but I'm trying to think about how people use their machines. I really don't want to lose the ability to quickly load a project from a different location.

For example: I might keep the major stash of projects on a USB drive I plug in because I don't have enough room on my laptop (based on a true story). Then I might also have a smaller stash of projects I use often on my small SSD hard drive.

Right now I can open projects wherever they are: box, usb , downloads folder, whatever. Could be really annoying to have to change the root folder every time I want to load a project from a different area.

Minor change:

If we do implement this I'd also add an extra folder level if I were you. The case I'm thinking of is that I have two identical projects that ran a day apart side by side. If you just use the project name you lose the context of the folder name you chose and thus, the ability to tell them apart. Also project names have no guarantee to identify a project in any kind of unique way.

so if the projects look like this on the file system:

/catalogdir/BRAT/17060304BRAT_yesterday_run/project.rs.xml
/catalogdir/BRAT/17060304BRAT_today_run/project.rs.xml

They should look like this in the catalog tree:

-ROOT(folder)
    -BRAT(folder)
        -17060304BRAT_yesterday_run(folder)
                -BRAT Run for HUC 17060304    <-------This is what I can select and load
        -17060304BRAT_today_run(folder)
                -BRAT Run for HUC 17060304    <-------This is what I can select and load

In summary:

1. Nothing disqualifying here so I'm definitely not saying we shouldn't implement a project catalog feature
2. but maybe we could load projects from disk too?
3. We need to think carefully about edge cases, as usual. This kind of problem always gets hairy on the edges.
   • eventually we may end up with a nest-y folder with thousands of projects. Can we scrape this fast enough that the UI doesn't become unusable?
   • Think carefully about things like duplicate projects and the messy and different ways people organize their hard drives.
   • Symlinks? Mounted drives? Cross platform concerns? Network drives? Slow connections? disconnected drives? File locking on windows?

[philipbaileynar]

All good feedback.

Clarifications:

1. ArcCatalog let's you connect to multiple folders. So you could have several stashes in variations locations.
2. We would definitely still show folders (per your project names might be the same).
3. .net has a fast method for find all files that match a pattern (*.project.rs.xml) recursively. Worked super fast for champ.

[MattReimer]

This has been on my mind a lot. Something about it just doesn't sit right with me. So here are a few more reasons I think it's not great:

• Moving multi-Gb projects takes time. If all I want to do is open a project quickly it's another minute of copying/moving AFTER downloading before I can just see it. That's a lot of scrollbars before I can see my data.
• It's an opinion about how people organize their hard drives but it doesn't have a really good reason behind it (my opinion). We should be really careful about telling people how to organize files unless we're sure it's right.
• QRave up until this point has not cared about anything outside a project folder. Not saying it has to be perfectly agnostic forever but if we're venturing outside the project folder we should do it slowly and carefully and for the right reasons. When we get a warehouse search and download functionality we will definitely have to have a place to download our projects and when that day comes I don't want to be limited by decisions we make here.
• Window management is terrible in Qt. I really don't want to have another kind of window/panel/dialog to move around that could potentially get lost/hidden or not behave.

Ok, enough being negative. What's the answer?

I think the answer has actually been sitting under our noses the whole time: we have a perfectly usable, accessible browser panel that most people will have open anyway. Why not just extend it to include a Riverscapes item that expands to projects.

For the record I'm suggesting an implementation that is exactly what @philipbaileynar suggested. I'm just saying let's do it in the browser and leave the open button alone.

The MapTiler plugin does it and their plugin is open source so we can "borrow" it:

Take a look at the source code here.

https://github.com/maptiler/qgis-maptiler-plugin/blob/master/browser_mapitem.py

Advantages of extending Browser:

1. We're using something that exists instead of inventing an arbitrary system. If people use the browser they will know how to work this. If they don't, nothing changes for them.
2. It's not another panel or dialog that people need to track. It's just there
3. It's additive. We're not taking anything away from the user or to do anything locally if they don't want to.

I'm not sure if this is going to be do-able in Arc but if not maybe this is one of those cases where the two plugins can diverge a bit.

Let's keep the dialog going. I'm happy to discuss any of this. It's about how people load their data though so I think a little caution here is definitely warranted.

[MattReimer]

Also finally, (and I promise this is the last thing) I was looking at the documentation for QFileDialog class and I think it should be possible to make some changes to make our stock file dialog more friendly. Not sure all of this is possible but things we could look into are:

1. Currently project.rs.xml files are the only thing you can select. We might reinforce that idea by changing their icon to the riverscapes icon and having a hover over tooltip or something that says "This is a riverscapes project file"
2. We could look at doing the same treatment as (1) but making it a folder browser instead of a file browser where the only folders you can select are ones with project files in them. We might be able to give those folders a different icon. (investigation needed here of course)

Source: https://doc.qt.io/qt-5/qfiledialog.html

[joewheaton]

Hey @MattReimer thanks for your thoughtful ideas here. I like the addition of using the Browser thing as a custom type (not sure an analogous thing is possible with ESRI's ArcCatalog. I agree it is a good partial solution for QGIS, but I think the most compelling thing might be to do your "Folder Browser" suggestion instead of browsing for a file. So all you're doing is browsing for a folder that has been filtered to have a project.rs.xml in it? It is actually the folders that have reasonable names (somewhat different from the name tag in metadata, but pretty good and it seems like that could work in ArcRAVE too? @philipbaileynar reactions?

[MattReimer]

@joewheaton I can't say for 100% certain that the folder browser idea is going to work. I'm only a few minutes into this. I did notice some good signals and slots on the file dialog that lend themselves to customizing the experience a little (that's the key here I think: as little as possible). I think we might need to try it first and see if it's any good.

[joewheaton]

@joewheaton I can't say for 100% certain that the folder browser idea is going to work. I'm only a few minutes into this. I did notice some good signals and slots on the file dialog that lend themselves to customizing the experience a little (that's the key here I think: as little as possible). I think we might need to try it first and see if it's any good.

Understood. Like 17 years ago I know you could do a folder browser control in .NET as I did one for my landscape evolution model GUI. I forgot that was how I handled this problem. So even if it does not work on Q side, might be worth exploring on the Arc side. We'll see how she goes.

[philipbaileynar]

The folder browser dialog in Windows is the WORST form in the history of human kind. You can select a folder, but you can't scroll to it, which essentially makes the form useless. It also has no ability to filter items or any other kind of interactivity.

Also, the browser might be customizable in QGIS but ArcCatalog almost certainly is not.

[joewheaton]

This is still an important issue to me.

[MattReimer]

Now that QRiS is a thing in the world we should include it in this conversation.... when we have it.