AL-850: adding tweakreg into stcal (#267)

* moved tweakreg into stcal, including docs and tests

* removed implicit dependency on datamodel structures and associated protocol

* added dependency on requests and tweakwcs modules

* added some alignment utils and modified existing ones, mostly to remove datamodel dependency

* ensured wcsinfo can be passed as a dict into all functions, documented what keys are required in wcsinfo

* run mypy in ci

---------

Co-authored-by: Brett <brettgraham@gmail.com>
Co-authored-by: Nadia Dencheva <nadia.astropy@gmail.com>

.github/workflows/ci.yml

@@ -23,6 +23,7 @@ jobs:
     with:
       envs: |
         - linux: check-style
+        - linux: check-types
         - linux: check-build
   test:
     uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1

.gitignore

@@ -74,6 +74,11 @@ instance/
 # Sphinx documentation
 docs/_build/
 
+# auto-generated API docs
+docs/source/api
+docs/api
+.DS_Store
+
 # PyBuilder
 .pybuilder/
 target/
@@ -146,9 +151,5 @@ src/stcal/ramp_fitting/ols_cas22/*.html
 # setuptools-scm generated module
 src/stcal/_version.py
 
-# auto-generated API docs
-docs/source/api
-.DS_Store
-
 # VSCode stuff
 .vscode

CHANGES.rst

@@ -1,10 +1,10 @@
-1.7.4 (unreleased)
+1.8.0 (unreleased)
 ==================
 
 General
 -------
 
--
+- Add TweakReg submodule. [#267]
 
 Changes to API
 --------------

docs/conf.py

@@ -47,8 +47,11 @@ def setup(app):
     "matplotlib": ("https://matplotlib.org/stable", None),
     "gwcs": ("https://gwcs.readthedocs.io/en/latest/", None),
     "astropy": ("https://docs.astropy.org/en/stable/", None),
+    "tweakwcs": ("https://tweakwcs.readthedocs.io/en/latest/", None),
 }
 
+nitpick_ignore = [("py:class", "optional"), ("py:class", "np.ndarray")]
+
 extensions = [
     "pytest_doctestplus.sphinx.doctestplus",
     "sphinx.ext.autodoc",

docs/stcal/alignment/description.rst

@@ -2,3 +2,21 @@ Description
 ============
 
 This sub-package contains all the modules common to all missions.
+
+WCS Info Dictionary
+-------------------
+Many of the functions in this submodule require a ``wcsinfo`` dictionary.
+This dictionary contains information about the spacecraft pointing, and
+requires at least the following keys:
+
+- ``'ra_ref'``: The right ascension at the reference point in degrees.
+- ``'dec_ref'``: The declination at the reference point in degrees.
+- ``'v2_ref'``: The V2 reference point in arcseconds,
+  with ``'v3_ref'`` maps to ``'ra_ref'`` and ``'dec_ref'``.
+- ``'v3_ref'``: The V3 reference point in arcseconds,
+  with ``'v2_ref'`` maps to ``'ra_ref'`` and ``'dec_ref'``.
+- ``'roll_ref'``: Local roll angle associated with each aperture in degrees.
+- ``'v3yangle'``: The angle between V3 and North in degrees.
+- ``'vparity'``: The "V-parity" of the observation, which is 1 if the
+  V3 axis is parallel to the detector Y-axis, and -1 if the V3 axis is
+  parallel to the detector X-axis.

docs/stcal/package_index.rst

@@ -7,3 +7,4 @@ Package Index
    jump/index.rst
    ramp_fitting/index.rst
    alignment/index.rst
+   tweakreg/index.rst

docs/stcal/tweakreg/astrometric_utils.rst

@@ -0,0 +1,12 @@
+=================
+astrometric_utils
+=================
+
+The ``astrometric_utils`` module provides functions for generating
+astrometric catalogs of sources for the field-of-view covered by a
+set of images.
+
+.. currentmodule:: stcal.tweakreg.astrometric_utils
+
+.. automodapi:: stcal.tweakreg.astrometric_utils
+   :noindex:

docs/stcal/tweakreg/description.rst

@@ -0,0 +1,228 @@
+Description
+===========
+
+Overview
+--------
+This step takes as input image catalogs of point-like sources, 
+which are used to compute corrections to the WCS of
+the input images such that sky catalogs obtained from the
+image catalogs using the corrected WCS will align on the sky.
+
+Source Catalogs
+---------------
+The input catalog must be in a format automatically recognized by
+`~astropy.table.Table.read`. The catalog must contain
+either ``'x'`` and ``'y'`` or ``'xcentroid'`` and ``'ycentroid'`` columns which
+indicate source *image* coordinates (in pixels). Pixel coordinates are
+0-indexed. An optional column in the catalog is the ``'weight'`` column,
+which when present, will be used in fitting.
+
+Relative Alignment
+------------------
+Relative alignment is performed by
+the `~stcal.tweakreg.relative_align` function.
+The source catalogs for each input image are compared to each other
+and linear (affine) coordinate transformations that align these
+catalogs are derived.  This fit ensures that all the input images
+are aligned relative to each other.  This step produces a combined
+source catalog for the entire set of input images as if they were
+combined into a single mosaic.
+
+Absolute Alignment
+------------------
+
+Absolute alignment is performed by
+the `~stcal.tweakreg.absolute_align` function.
+If the parameter ``abs_refcat`` is set to 'GAIADR3', 'GAIADR2', or 'GAIADR1',
+an astrometric reference catalog then gets generated by querying
+a GAIA-based astrometric catalog web service for all astrometrically
+measured sources in the combined field-of-view of the set of input
+images. This catalog is generated from the catalogs available
+through the `STScI MAST Catalogs`_ and has the ability to account
+for proper motion to a given epoch. The epoch is computed from the observation date and time
+of the input data.
+
+.. _STScI MAST Catalogs: https://outerspace.stsci.edu/display/MASTDATA/Catalog+Access
+
+The combined source catalog 
+then gets cross-matched and fit to this astrometric reference catalog.
+The pipeline initially supports fitting to the
+GAIADR3 catalog, with the option to select the GAIADR2 or GAIADR1 instead.
+The results of this one fit then gets back-propagated to all the
+input images to align them all to the astrometric reference frame while
+maintaining the relative alignment between the images.
+
+For this part of alignment, instead of 'GAIADR1', 'GAIADR2', or 'GAIADR3', users can
+supply an external reference catalog by providing a path to an existing
+file. A user-supplied catalog must contain ``'RA'`` and ``'DEC'`` columns
+indicating reference source world coordinates (in degrees). An optional column
+in the catalog is the ``'weight'`` column, which when present, will be used
+in fitting. The catalog must be in a format automatically recognized by
+`~astropy.table.Table.read`.
+
+Grouping
+--------
+
+Images taken at the same time (e.g., NIRCam images from all short-wave
+detectors) can be aligned together; that is, a single correction
+can be computed and applied to all these images because any error in
+telescope pointing will be identical in all these images and it is assumed
+that the relative positions of (e.g., NIRCam) detectors do not change.
+Identification of images that belong to the same "exposure" and therefore
+can be grouped together is based on several attributes 
+*change from here to end of section* described in
+`~jwst.datamodels.ModelContainer`. This grouping is performed automatically
+in the ``tweakreg`` step using the
+`~jwst.datamodels.ModelContainer.models_grouped` property, which assigns
+a group ID to each input image model in ``meta.group_id``.
+
+However, when detector calibrations are not accurate, alignment of groups
+of images may fail (or result in poor alignment). In this case, it may be
+desirable to align each image independently. This can be achieved either by
+setting the ``image_model.meta.group_id`` attribute to a unique string or integer
+value for each image, or by adding the ``group_id`` attribute to the ``members`` of the input ASN
+table - see `~jwst.datamodels.ModelContainer` for more details.
+
+.. note::
+    Group ID (``group_id``) is used by both ``tweakreg`` and ``skymatch`` steps
+    and so modifying it for one step will affect the results in another step.
+    If it is desirable to apply different grouping strategies to the ``tweakreg``
+    and ``skymatch`` steps, one may need to run each step individually and
+    provide a different ASN as input to each step.
+
+WCS Correction
+--------------
+The linear coordinate transformation computed in the previous step
+is used to define tangent-plane corrections that need to be applied
+to the GWCS pipeline in order to correct input image WCS.
+This correction is implemented by inserting a ``v2v3corr`` frame with
+tangent plane corrections into the GWCS pipeline of the image's WCS.
+
+Arguments
+---------
+The ``tweakreg`` step has the following optional arguments:
+
+
+**Relative alignment parameters:**
+
+Parameters used for relative alignment between input images. These parameters
+are passed into the `~stcal.tweakreg.relative_align` function.
+
+* ``expand_refcat``: A boolean indicating whether or not to expand reference
+  catalog with new sources from other input images that have been already
+  aligned to the reference image. (Default=False)
+
+* ``minobj``: A positive `int` indicating minimum number of objects acceptable
+  for matching. (Default=15)
+
+* ``searchrad``: A `float` indicating the search radius in arcsec for a match.
+  (Default=2.0)
+
+* ``use2dhist``: A boolean indicating whether to use 2D histogram to find
+  initial offset. (Default=True)
+
+* ``separation``: Minimum object separation in arcsec. It **must be** at least
+  ``sqrt(2)`` times larger than ``tolerance``. (Default=1.0)
+
+* ``tolerance``: Matching tolerance for ``xyxymatch`` in arcsec. (Default=0.7)
+
+* ``xoffset``: Initial guess for X offset in arcsec. (Default=0.0)
+
+* ``yoffset``: Initial guess for Y offset in arcsec. (Default=0.0)
+
+* ``fitgeometry``: A `str` value indicating the type of affine transformation
+  to be considered when fitting catalogs. Allowed values:
+
+  - ``'shift'``: x/y shifts only
+  - ``'rshift'``: rotation and shifts
+  - ``'rscale'``: rotation and scale
+  - ``'general'``: shift, rotation, and scale
+
+  The default value is "rshift".
+
+  .. note::
+      Mathematically, alignment of images observed in different tangent planes
+      requires ``fitgeometry='general'`` in order to fit source catalogs
+      in the different images even if mis-alignment is caused only by a shift
+      or rotation in the tangent plane of one of the images.
+
+      However, under certain circumstances, such as small alignment errors or
+      minimal dithering during observations that keep tangent planes of the
+      images to be aligned almost parallel, then it may be more robust to
+      use a ``fitgeometry`` setting with fewer degrees of freedom such as
+      ``'rshift'``, especially for "ill-conditioned" source catalogs such as
+      catalogs with very few sources, or large errors in source positions, or
+      sources placed along a line or bunched in a corner of the image (not
+      spread across/covering the entire image).
+
+* ``nclip``: A non-negative integer number of clipping iterations
+  to use in the fit. (Default=3)
+
+* ``sigma``: A positive `float` indicating the clipping limit, in sigma units,
+  used when performing fit. (Default=3.0)
+
+**Absolute Astrometric fitting parameters:**
+
+Parameters used for absolute astrometry to a reference catalog.
+These parameters
+are passed into the `~stcal.tweakreg.absolute_align` function.
+
+* ``abs_refcat``: String indicating what astrometric catalog should be used.
+  Currently supported options: 'GAIADR1', 'GAIADR2', 'GAIADR3', a path to an existing
+  reference catalog, `None`, or `''`. See
+  `stcal.tweakreg.tweakreg.SINGLE_GROUP_REFCAT`
+  for an up-to-date list of supported built-in reference catalogs.
+
+  When ``abs_refcat`` is `None` or an empty string, alignment to the
+  absolute astrometry catalog will be turned off.
+  (Default= `''`)
+
+* ``abs_minobj``: A positive `int` indicating minimum number of objects
+  acceptable for matching. (Default=15)
+
+* ``abs_searchrad``: A `float` indicating the search radius in arcsec for
+  a match. It is recommended that a value larger than ``searchrad`` be used for
+  this parameter (e.g. 3 times larger) (Default=6.0)
+
+* ``abs_use2dhist``: A boolean indicating whether to use 2D histogram to find
+  initial offset. It is strongly recommended setting this parameter to `True`.
+  Otherwise the initial guess for the offsets will be set to zero
+  (Default=True)
+
+* ``abs_separation``: Minimum object separation in arcsec. It **must be** at
+  least ``sqrt(2)`` times larger than ``abs_tolerance``. (Default=1.0)
+
+* ``abs_tolerance``: Matching tolerance for ``xyxymatch`` in arcsec.
+  (Default=0.7)
+
+* ``abs_fitgeometry``: A `str` value indicating the type of affine
+  transformation to be considered when fitting catalogs. Allowed values:
+
+  - ``'shift'``: x/y shifts only
+  - ``'rshift'``: rotation and shifts
+  - ``'rscale'``: rotation and scale
+  - ``'general'``: shift, rotation, and scale
+
+  The default value is "rshift". Note that the same conditions/restrictions
+  that apply to ``fitgeometry`` also apply to ``abs_fitgeometry``.
+
+* ``abs_nclip``: A non-negative integer number of clipping iterations
+  to use in the fit. (Default=3)
+
+* ``abs_sigma``: A positive `float` indicating the clipping limit, in sigma
+  units, used when performing fit. (Default=3.0)
+
+* ``save_abs_catalog``: A boolean specifying whether or not to write out the
+  astrometric catalog used for the fit as a separate product. (Default=False)
+
+Further Documentation
+---------------------
+The underlying algorithms as well as formats of source catalogs are described
+in more detail at
+
+https://tweakwcs.readthedocs.io/en/latest/
+
+.. currentmodule:: stcal.tweakreg.tweakreg
+
+.. automodapi:: stcal.tweakreg.tweakreg
+   :noindex:

docs/stcal/tweakreg/index.rst

@@ -0,0 +1,20 @@
+.. _tweakreg_step:
+
+========
+TweakReg
+========
+
+.. toctree::
+   :maxdepth: 2
+
+   description.rst
+
+**Also See:**
+
+.. toctree::
+   :maxdepth: 1
+
+   utils.rst
+   astrometric_utils.rst
+
+.. automodapi:: stcal.tweakreg

docs/stcal/tweakreg/utils.rst

@@ -0,0 +1,11 @@
+=================
+Utility Functions
+=================
+
+Currently, the ``utils`` module provides helpful functions for
+manually applying corrections to an imaging WCS.
+
+.. currentmodule:: stcal.tweakreg.utils
+
+.. automodapi:: stcal.tweakreg.utils
+   :noindex:

pyproject.toml

@@ -19,6 +19,8 @@ dependencies = [
     "opencv-python-headless >=4.6.0.66",
     "asdf >=2.15.0",
     "gwcs >= 0.18.1",
+    "tweakwcs >=0.8.8",
+    "requests >=2.22",
 ]
 dynamic = [
     "version",
@@ -194,6 +196,26 @@ convention = "numpy"
 [tool.ruff.lint.flake8-annotations]
 ignore-fully-untyped = true  # Turn of annotation checking for fully untyped code
 
+[tool.mypy]
+python_version = "3.12"
+warn_return_any = true
+warn_unused_configs = true
+
+[[tool.mypy.overrides]]
+module = [
+    "astropy.*",
+    "gwcs.*",
+    "tweakwcs.*",
+    "stdatamodels.*",
+    "asdf.*",
+    "scipy.*",
+    # don't complain about the installed c parts of this library
+    "stcal.ramp_fitting.ols_cas22._fit",
+    "stcal.ramp_fitting.ols_cas22._jump",
+    "stcal.ramp_fitting.slope_fitter",
+]
+ignore_missing_imports = true
+
 [tool.cython-lint]
 max-line-length = 110