Support for third party instruments

[Alex-J-Brown]

As a well-featured photometry pipeline it would be really useful to be able to reduce data from other imagers without having to go through the process of translating each frame into a .hcm file which is quite messy in terms of the number of files made and can also be quite inefficient with regards to storage if the raw data is in a fits cube format. This is especially important for modern CMOS imagers such as the ORCA-QUEST2 which can produce a significant amount of data in a short time and therefore having to extract a fits cube into thousands of individual .hcm files in order to be able to reduce the data can be a real problem. Pipeline support to be able to reduce the data directly from the native output format of such instruments would therefore be a really useful addition.

[vikdhillon]

Kevin Burdge (MIT) made a similar request to me last year. He also has an ORCA-QUEST2, which he uses on the 5m Hale and 6.5m Magellan Telescopes. He uses the HiPERCAM pipeline for all his time-series photometric reduction and would very much like the pipeline to support his camera's data format directly rather than having to run fits2hcm first.

[StuartLittlefair]

Great suggestion - we should implement this as soon as resources allow. Since all of these formats are essentially multi-extension FITS files, with each extension representing a new frame, the best approach is probably to add a framework that allows anyone to add an instrument to the pipeline. We would create two new abstract base classes:

MEFHead - reads in the header info in HDUL[0] and creates the window formats needed for this camera;
MEFData - basically a callable class which returns the next frame as an MCCD object.

examples for these classes can be seen in hcam.py, they are analogs of Rdata and Rhead

Anyone wishing to add a new instrument can subclass these base classes and add code that is specific to their instrument - e.g which header items to select for information the pipeline needs, the window format of their data etc.

[StuartLittlefair]

Thinking a little further about how this might work, someone who wanted their instrument to work natively with the hipercam pipeline would write their own classes for the Rdata, Rhead and Rtbytes objects, hopefully with as much boilerplate moved to the base classes as possible.

For an instrument "X', the user could implement them in a new package hipercam-X, e.g:

class Rdata(MEFdataBase):
    pass

class Rhead(MEFheadBase):
    pass

class Rtybyes(MEFtbytesBase):
    pass

class Instrument:
    rdata: Rdata
    rhead: Rhead
    rtybyes: Rtbytes

We can use the entry points mechanism to register the new instrument, e.g in the pyproject.toml of the new package, we'd have:

[project.entry-points.'hipercam.instruments']
instX = 'hipercam_x.Instrument'

This is then discoverable by the hipercam pipeline, so in spooler.py we can refactor a bit and have something like this:

import sys
if sys.version_info < (3, 10):
    from importlib_metadata import entry_points
else:
    from importlib.metadata import entry_points

discovered_instruments = entry_points(group='hipercam.instruments')

def data_source(source):
     if source not in ('hl', 'hs', ....):
         # try and find source in plugins
         instrument =  discovered_instruments[source]
    else:
        # builtin
        ....