jump step for the uneven fitting requires huge amounts of memory

[stscijgbot-rstdms]

Issue RCAL-702 was created on JIRA by Jonathan Eisenhamer:

Using data found in:

{{██████████████████████████████████████████

running the command

strun romancal.step.RampFitStep random_dqinit.asdf

on a m1 mac uses ~ 150GB of memory and takes ~5 minutes to run. This occurs regardless of the value of use_jump_detection.

The same data/command, using commits pre-jump detection, results in a runtime < 5 secs and not long enough to see memory usage in the activity monitor.

[stscijgbot-rstdms]

Comment by William Jamieson on JIRA:

I think I what is causing this, the code is retaining all of the individual fits from splitting the ramps across the dqflags in the case or jumps in the general case. The previous code was not doing this (which is entirely reasonable), and this change was added by me in order to get full diagnostic data on every fit made.

Eddie Schlafly should we bother retaining the ability to retain this information for the future. This can be done with an optional argument, otherwise I can simply remove this.

An additional contributing factor is that it now sets the DQ flags for jump detection on every resultant, which we currently don’t have as part of the rampfit output model.

[schlafly]

Remind me: every fit result is something like 4 floats (slope, uncertainties) for certainly no more than the number of resultants, so this should make a maximum of 4 * 4 bytes * 4088^2 * 6? I'm missing some large factor! The old ramp code also had the individual segment fits in memory before averaging them down (split by the old jump detection), but that wasn't a big hit. It's entirely possible that something's gone wrong with the romanisim / romancal interaction or something and we're splitting every resultant into as many segments as possible, but this number still seems like a lot.

[stscijgbot-rstdms]

Comment by Nadia Dencheva on JIRA:

Please close this if fixed with a link to the PR.

[stscijgbot-rstdms]

Comment by William Jamieson on JIRA:

Closed by: spacetelescope/stcal#226