Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RCAL-702: Fix memory use issue for Cas22 jump detection #226

Merged
merged 4 commits into from
Nov 6, 2023

Conversation

WilliamJamieson
Copy link
Contributor

Resolves RCAL-702

This PR resolves a massive memory spike as part of the solution to RCAL-702

Checklist

  • added entry in CHANGES.rst (either in Bug Fixes or Changes to API)
  • updated relevant tests
  • updated relevant documentation
  • updated relevant milestone(s)
  • added relevant label(s)

Copy link

codecov bot commented Nov 1, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Files Coverage Δ
src/stcal/ramp_fitting/ols_cas22_fit.py 81.25% <100.00%> (ø)
tests/test_jump_cas22.py 100.00% <100.00%> (ø)

📢 Thoughts on this report? Let us know!.

Copy link
Collaborator

@schlafly schlafly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks!

@schlafly
Copy link
Collaborator

schlafly commented Nov 1, 2023

@stscieisenhamer , mind checking to see if this resolves your issue? Thanks!

Copy link
Collaborator

@stscieisenhamer stscieisenhamer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dramatic change in memory usage: From 150GB to not even breaking 1GB.

And also, there is an actual run-time difference in running with and without --use_jump_detection. Without the step runs ~1.5 minutes. With jump detection, the task runs about as long as before this fix, ~6 minutes. As a reminder, before the runtime was no different between the two.

LGTM

@schlafly
Copy link
Collaborator

schlafly commented Nov 2, 2023

I think I see that with include_diagnostic=False we don't get any information back about where the jumps are and we don't end up updating the DQ array. I think we need that. Ideally that could happen in place on the DQ array, so no additional memory were needed; I'm not sure how convenient that ends up being here.

@schlafly schlafly merged commit a4849f7 into spacetelescope:main Nov 6, 2023
25 checks passed
@WilliamJamieson WilliamJamieson deleted the bugfix/jump_memory branch November 7, 2023 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants