Results files in the repo

[tsmbland]

I think it's worth thinking again about which files we're committing to the repo.

Currently we're committing all results files from all the example/tutorial models. This means that every time any of these models change, we get a whole load of changes (currently 25,000 additions and deletions in #349), and I don't think it's sustainable to keep committing all these changes to the repo. It's already 1.5 GB and growing.

As far as I can tell, the only reason that we need to commit the results files is for the regression tests. Would it not be better to create some kind of checksum? We'd have to be a bit careful because we often get subtly different results on different operating systems, so we'd need the checksum to be tolerant to some degree of error, but there must be some solution that doesn't involve committing the full results files...

@dalonsoa @alexdewar

[alexdewar]

I agree. It's not good to be changing 20k lines with every PR, for multiple reasons.

Thinking about it, we could change the regression tests so that you can pass in a path to some old results (with an env var or something), then we could drop the results from the repo. We can do some trickery with our CI workflows to first generate the old results from whatever version of MUSE is on the commit the PR is based on (e.g. the tip of develop) first. (We can even cache the results, if we're feeling fancy.) We could also upload generated results with the documentation, if that's needed.

[tsmbland]

Yeah (well that's what I've been doing at least)

[dalonsoa]

I agree that this is not sustainable, but I prefer the idea of keeping a file with checksums - platform dependent, if needed. Pointing to a specific hash feels a bit hacky and I think it might fail in forks or if history needs clean-up (as we might need to do to get rid of 1.5 Gb of useless data).

[alexdewar]

But if we did rebase the entire repo, we'd still only need to change that one commit hash rather than checksums for all of the individual files. I agree it's a bit hacky, but if we can relegate the generation/caching of regression data to the CI then it will make things easier.

Although the results would differ between platforms (e.g. mac ARM would be different), the bigger problem is that you'll get different results for different CPU types, so you would get different results on Intel and AMD processors, even though they're both x86. It also means if you're running this on x86 macOS or ARM Linux (ok, this one isn't too likely), then the tests will also fail. I think having a tolerance for floating point error is the right way to do this.

[alexdewar]

PS -- I just double-checked and the .git folder in my repo is ~40MB, which isn't terrible. Maybe you were also including your .venv in the count @tsmbland?

Either way we should fix this, but we probably don't need to retroactively rip all the results files out of the repo.

[tsmbland]

Ahh you're right - silly me!

[tsmbland]

I'm going to close this. The outputs are now much smaller than they once were, so I'm less worried about the repository booming. I also find it quite useful committing the results files as it's easy to see exactly what's changed in the diffs, which you wouldn't get with a checksum