Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-43712: Add configurable buffer for template and refcat preload #204

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Commits on Sep 27, 2024

  1. Update unit tests with the test data change

    This should have been together with 674a4b1
    
    Unlike the old DECam test dataset, the new test LSSTComCamSim dataset
    does not have crosstalk. So, use another dataset type to test.
    
    Also fix a time stamp missed in the previous commit.
    hsinfang committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    30770f0 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Fix a missing space

    hsinfang committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    6e01665 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0bb8cb6 View commit details
    Browse the repository at this point in the history
  3. Make the preload padding configurable at the service level

    The optimal value likely depends on the instrument and the
    skymap choice.
    hsinfang committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    67bd7f1 View commit details
    Browse the repository at this point in the history
  4. Use the instrument's formatter to get the sky wcs

    Instead of determining the instrument's wcsFlipX here, it is more
    robust to use its formatter, which knows its camera orientation
    from its obs package.
    hsinfang committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    898fa56 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2696a76 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Configuration menu
    Copy the full SHA
    f97422c View commit details
    Browse the repository at this point in the history
  2. Use regionTimeInfo's region to export template/refcat datasets

    This changes the APIs of _export_skymap_and_templates and
    _export_refcats to take a lsst.sphgeom.Region directly.
    
    Note that the centroid of the region is not the same of the
    detector center, but it should not matter.
    
    Because htm7 can be too coarse compared to the patch size, using
    htm7 indices to search for templates may lead to preloading more
    patches than necessary and wasting time. This feature of using
    htm7 to search for overlapping templates is also about to be
    deprecated and replaced by the arbitrary spatial region query in
    Butler. The usage will be replaced when switching to the new
    butler query system.
    hsinfang committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    407b263 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    eff9ca6 View commit details
    Browse the repository at this point in the history
  4. Query one datasetType at a time for calibs/refcats/templates

    This is a preparation step before switching to the new butler query
    system.
    
    In the new query system, query_datasets takes one datasetType at a
    time, while butler.registry.queryDatasets can take a list of dataset
    types in the old system. So, we change to query (via _filter_datasets)
    one type of calibs/refcats/templates at a time.
    
    Currently we can preload more types of calibs/refcats/templates
    than the actual pipelines really need. It's possible that some types
    are not preloaded but it's okay. For now we allow _MissingDatasetError.
    hsinfang committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    057e256 View commit details
    Browse the repository at this point in the history
  5. Use the new Butler query system in _filter_datasets

    The new Butler query systems supports spatial-constraint query
    via lsst.sphgeom.Region directly. With this change, we use it
    in template and refcat search. This needs stack w_2024_38 or newer.
    
    make_export.py uses _filter_datasets so it needs to adjust to the
    new underlying API too.
    hsinfang committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    c6860f8 View commit details
    Browse the repository at this point in the history
  6. Enable prep_butler unit tests again

    Some unit tests were temporarily marked expectedFailure in 674a4b1.
    
    Now that we switch to the new query system, make them work again.
    
    The test repo was put together using middleware tools, which
    intrinsically uses butler repo's visit-detector regions with its
    padding from defineVisits config.
    
    That padding config is not the same as the preload region padding
    in prompt processing.
    
    This explains the patch differences in template selection.
    hsinfang committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    5653980 View commit details
    Browse the repository at this point in the history