example-run.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example pipeline execution\n",
    "\n",
    "Here is an example of what it takes to configure and run a pipeline with flat-fielding and background subtraction.\n",
    "\n",
    "## Setup CRDS\n",
    "\n",
    "Make sure you have a local checkout of the CRDS cache as explained in the [Getting started page](getting-started.rst). Either run `source setup_local_crds.sh` to setup the enviroment variables in the shell or execute `setup_local_crds.py` for Python. This is needed to point the CRDS software to the CRDS cache. Optionally source this in your shell configuration to automatically set this up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run ../setup_local_crds.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get input simulations\n",
    "Download simulated input FITS files for the IRIS imager from [Figshare](https://figshare.com/articles/TMT_IRIS_test_simulations/9941939). It contains a raw science frame, a raw flat frame and a raw background frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import zipfile\n",
    "import os.path\n",
    "\n",
    "if not os.path.exists(\"iris_example_data/raw_flat_frame_cal.fits\"):\n",
    "    urllib.request.urlretrieve(\n",
    "        \"https://ndownloader.figshare.com/articles/9941939/versions/1\",\n",
    "        \"iris_example_data.zip\"\n",
    "    )\n",
    "\n",
    "    with zipfile.ZipFile(\"iris_example_data.zip\", 'r') as zip_ref:\n",
    "        zip_ref.extractall(\"iris_example_data\")\n",
    "\n",
    "# this could take a few minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%ls iris_example_data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess the flat frame\n",
    "First we need to remove the dark frame from the flat frame and normalize it. A dark frame is already available in the CRDS and the pipeline knows how to retrieve it based on the metadata in the FITS file headers.\n",
    "\n",
    "We can check in the package documentation what are the available pipelines and check the configuration options of the pipeline.ProcessFlatfieldL2 class.\n",
    "\n",
    "We do not need to customize it so we can directly call it from tmtrun and pass the input FITS file:\n",
    "\n",
    "This will pickup the relevant dark frame from the CRDS and process the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tmtrun iris_pipeline.pipeline.ProcessFlatfieldL2 iris_example_data/raw_flat_frame_cal.fits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the image processing pipeline\n",
    "\n",
    "The [`ProcessImagerL2Pipeline`](api/iris_pipeline.pipeline.ProcessImagerL2Pipeline.rst#iris_pipeline.ProcessImagerL2Pipeline) can be configured using a INI-style configuration file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file image2_iris.cfg\n",
    "\n",
    "\n",
    "name = \"ProcessImagerL2Pipeline\"\n",
    "class = \"iris_pipeline.pipeline.ProcessImagerL2Pipeline\"\n",
    "save_results = True\n",
    "\n",
    "    [steps]\n",
    "      [[bkg_subtract]]\n",
    "      [[assign_wcs]]\n",
    "        skip = True\n",
    "      [[flat_field]]\n",
    "        config_file = flat_field.cfg\n",
    "      [[photom]]\n",
    "        skip = True\n",
    "      [[resample]]\n",
    "        skip = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "first we specify that we want to execute the pipeline defined in the pipeline, then we can configure each of the steps, for example skip some of them. Also we can import the configuration of a step from another file, in this case flat_field.cfg:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file flat_field.cfg\n",
    "\n",
    "name = \"flat_field\"\n",
    "class = \"jwst.flatfield.FlatFieldStep\"\n",
    "override_flat = 'raw_flat_frame_flat.fits'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the input data\n",
    "\n",
    "JWST created a specification for defining how input files should be used by a pipeline, it is a JSON file named an association, see the [JWST documentation](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/overview.html).\n",
    "\n",
    "In our example we need to specify a input raw science frame ad a background to be subtracted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file association.json\n",
    "\n",
    "{\n",
    "    \"asn_rule\": \"Asn_Lv2Image\",\n",
    "    \"asn_pool\": \"pool\",\n",
    "    \"asn_type\": \"image2\",\n",
    "    \"products\": [\n",
    "        {\n",
    "            \"name\": \"test_iris_subtract_bg_flat\",\n",
    "            \"members\": [\n",
    "                {\n",
    "                    \"expname\": \"iris_example_data/raw_science_frame_sci.fits\",\n",
    "                    \"exptype\": \"science\"\n",
    "                },\n",
    "                {\n",
    "                    \"expname\": \"iris_example_data/raw_background_frame_cal.fits\",\n",
    "                    \"exptype\": \"background\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute the pipeline from the command line\n",
    "\n",
    "Execute the pipeline from the command line\n",
    "We can use tmtrun from a terminal to execute the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tmtrun image2_iris.cfg association.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the reduced frame\n",
    "\n",
    "Finally we can use `iris_pipeline` to load the data and plot them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import iris_pipeline\n",
    "reduced_science_frame = iris_pipeline.datamodels.IRISImageModel(\"test_iris_subtract_bg_flat_cal.fits\")\n",
    "try:    \n",
    "    %matplotlib inline\n",
    "    import matplotlib.pyplot as plt\n",
    "    plt.figure()\n",
    "    plt.imshow(reduced_science_frame.data, vmin=0, vmax=1500)\n",
    "    plt.title(\"Processed frame\")\n",
    "\n",
    "except ImportError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm image2_iris.cfg association.json flat_field.cfg"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:jwst_dev]",
   "language": "python",
   "name": "conda-env-jwst_dev-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}