Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements and collect test improvements #107

Merged
merged 11 commits into from
Sep 11, 2024

Conversation

ZedThree
Copy link
Member

@ZedThree ZedThree commented Jul 31, 2024

Rebase of #101 to account for the moved source files

Fixes #68

Refactor of collect tests to make them run in a reasonable amount of time:

  • Cache dump files for each grid scenario
    • This gives a 25-30% speed-up
  • Remove some combinations of test parameters
    • The slicing test alone was almost 30% of the runtime due to the number of combinations
    • Reduced total number of tests from ~600 to ~200
  • Only make one of each kind of field in the test dump files
    • ~50% of the runtime of the tests was solely due to datafile.open, which we have to do at least once for every call to collect, sometimes more than 50 times
  • Reduce duplicated code by parameterising over the grid scenarios

Overall this reduces the time the tests take from >35 minutes to <2

dschwoerer and others added 11 commits July 31, 2024 17:17
We were testing slices in t, x, y, z, and all four combined. The
combined slices should really cover all the other cases, so we can get
rid of most of them. This just removes the y and z cases, but could
probably also remove the t and x ones too
This also changes it to just parameterise `mxg` and `myg` together
The majority of `collect` tests time is spent in `datafile.open`, and
because we re-open the file for each field we collect, reducing the
number of fields in the files gives us a big speed up
@mikekryjak
Copy link

This is very impressive!

@ZedThree ZedThree merged commit e458cf0 into master Sep 11, 2024
9 checks passed
@ZedThree ZedThree deleted the performance-improvements-rebase branch September 11, 2024 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unit tests are slow
3 participants