JP-3770: emicorr memory and run time improvement

[penaguerrero]

Resolves JP-3770

Closes #

This PR improves runtime and memory usage.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.79%. Comparing base (fb3c2cd) to head (2976d4b).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8849      +/-   ##
==========================================
- Coverage   61.79%   61.79%   -0.01%     
==========================================
  Files         377      377              
  Lines       38834    38832       -2     
==========================================
- Hits        23999    23997       -2     
  Misses      14835    14835              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tapastro]

Have regression tests been run?

[penaguerrero]

Reg tests started: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1762/

[penaguerrero]

Reg tests are all clear!

[drlaw1558]

Looks reasonable to me. Also tested out on jw01236004001_02101_00001_mirifulong_uncal.fits, 1 int and 188 groups. Before/after results are identical as expected. Emicorr runtime is also pretty much unchanged though, at 79.1 seconds before and 78.5 seconds after. What kind of runtime improvements were you finding on other test cases?

[penaguerrero]

@drlaw1558 more than runtime it is an improvement on the memory management, the runtime is a plus, especially over a large set of files to process. Try your test with file jw01386007001_04101_00006_mirimage_uncal.fits

[drlaw1558]

@penaguerrero This might be a case where the memory gains can have an outsize impact depending on the memory available to a given machine. I'm seeing 17 seconds in emicorr for jw01386007001_04101_00006_mirimage_uncal.fits both before and after the update, though I can believe this might be very different for machines that required using swap space a lot.

[melanieclarke]

Looks good to me! One quick request to add a little more detail to the change log, so we remember later what the improvement was.

[melanieclarke]

Can you add a little more detail here?

[penaguerrero]

Sure thing!