Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add select on read for MWA correlator FITS files #1458

Merged
merged 12 commits into from
Nov 1, 2024
Merged

Conversation

bhazelton
Copy link
Member

@bhazelton bhazelton commented Jul 6, 2024

Description

Add support for select on read to MWA correlator FITS files to save memory and time. This adds support for most but not all select keywords. The implementation is a bit complicated because of the structure of MWA correlator FITS files, which is also different between the files written by the legacy correlator and MWAX.

Edit: we have benchmarking now and the benchmarks didn't slow down significantly
I am a bit concerned that some of these changes may have slowed down the reading when not selecting on read. I think this needs to be tested/profiled more, would love some help on that front.

I also refactored some selection code for the baseline-time axis and the frequency axis into functions for better code reuse, leading to changes in UVData.select and UVCal.select. And I improved the way the history string is built up.

Motivation and Context

closes #1386

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation change (documentation changes only)
  • Version change
  • Build or continuous integration change

Checklist:

New feature checklist:

  • I have added or updated the docstrings associated with my feature using the numpy docstring format.
  • I have updated the tutorial to highlight my new feature (if appropriate).
  • I have added tests to cover my new feature.
  • All new and existing tests pass.
  • I have updated the CHANGELOG.

@bhazelton bhazelton force-pushed the mwa_corr_fits_select branch 2 times, most recently from cdfc0e3 to ae54798 Compare July 12, 2024 03:56
Copy link

codecov bot commented Jul 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.93%. Comparing base (05ee100) to head (f41752c).
Report is 12 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1458      +/-   ##
==========================================
- Coverage   99.93%   99.93%   -0.01%     
==========================================
  Files          62       63       +1     
  Lines       21756    21748       -8     
==========================================
- Hits        21741    21733       -8     
  Misses         15       15              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@mkolopanis mkolopanis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do feel like a lot of the mwa specific stuff is lost on me, but I do have a question after a quick read

src/pyuvdata/uvdata/mwa_corr_fits.py Show resolved Hide resolved
mkolopanis
mkolopanis previously approved these changes Oct 31, 2024
Copy link
Member

@mkolopanis mkolopanis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the little change. Looks good to me

@mkolopanis mkolopanis merged commit 4e6f3ed into main Nov 1, 2024
52 of 54 checks passed
@mkolopanis mkolopanis deleted the mwa_corr_fits_select branch November 1, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Select on read for MWA correlator fits
2 participants