From 65770230bcdd07ce2bab27fcbdc6f8e145607c17 Mon Sep 17 00:00:00 2001 From: Stephanie LaMassa Date: Tue, 23 Jan 2024 16:13:34 -0500 Subject: [PATCH 1/8] adding NIRISS imaging tutorial notebook --- .../niriss-imaging-tutorial.ipynb | 819 ++++++++++++++++++ 1 file changed, 819 insertions(+) create mode 100755 notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb new file mode 100755 index 000000000..a8c705d61 --- /dev/null +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -0,0 +1,819 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0393e357-9d9d-4516-b28d-4d335fad33a0", + "metadata": {}, + "source": [ + "# Baseline: NIRISS Imaging Tutorial Notebook\n", + "\n", + "## Table of Contents\n", + "1. [Introduction](#intro)\n", + "2. [Examining uncalibrated data products](#examine)\n", + "3. [Stage 1 Processing](#det1)\n", + "4. [Stage 2 Processing](#image2)\n", + "5. [Stage 3 Processing](#image3)\n", + "6. [Visualize Detected Sources](#detections)" + ] + }, + { + "cell_type": "markdown", + "id": "dc6b2aba-072e-4179-a856-29d547f277d6", + "metadata": {}, + "source": [ + "# 1. Introduction\n", + "In this notebook, we will process a [NIRISS imaging dataset](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-observing-modes/niriss-imaging) through the [JWST calibration pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview). The example dataset is from [Program ID](https://www.stsci.edu/jwst/science-execution/program-information) 1475 (PI: Boyer, CoI: Volk) which is a sky flat calibration program. NIRCam is used as the primary instrument with NIRISS as a [coordinated parallel instrument](https://jwst-docs.stsci.edu/methods-and-roadmaps/jwst-parallel-observations/jwst-coordinated-parallels-roadmap). The NIRISS imaging dataset uses a 17-step dither pattern.\n", + "\n", + "For illustrative purposes, we focus on data taken through the NIRISS [F150W filter](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-instrumentation/niriss-filters) and start with uncalibrated data products. The files are named\n", + "`jw01475006001_02201_000nn_nis_uncal.fits`, where *nn* refers to the dither step number which ranges from 01 - 17. See the [\"File Naming Schemes\" documentation](https://jwst-pipeline.readthedocs.io/en/stable/jwst/data_products/file_naming.html) to learn more about the file naming convention.\n", + "\n", + "In this notebook, we assume all uncalibrated fits files are saved in a directory named `1475_f150w`." + ] + }, + { + "cell_type": "markdown", + "id": "026a649d-a0d2-4e3f-ab6d-cf8c26e6ce1d", + "metadata": {}, + "source": [ + "## Install pipeline and dependencies" + ] + }, + { + "cell_type": "markdown", + "id": "9c5b415a-f0f4-4c48-a038-d319e825ee4f", + "metadata": {}, + "source": [ + "Make sure the [latest version of the pipeline is installed](https://jwst-pipeline.readthedocs.io/en/stable/index.html) and that the [Calibration Reference Data System (CRDS)](https://jwst-crds.stsci.edu/static/users_guide/index.html) is configured so the most up-to-date calibration files are used to process the dataset. This notebook was created with pipeline version `1.12.3`. \n", + "\n", + "This notebook uses the [`jdaviz` package](https://jdaviz.readthedocs.io/en/latest/index.html) to visualize the datasets. To [install `jdaviz`](https://jdaviz.readthedocs.io/en/latest/installation.html) in your current environment:\n", + "```\n", + "pip install jdaviz --upgrade\n", + "```\n", + "\n", + "## Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb9f7f54-ff98-428b-9fa9-b39c50210c3a", + "metadata": {}, + "outputs": [], + "source": [ + "import os, glob\n", + "import numpy as np\n", + "\n", + "# Configure CRDS\n", + "os.environ[\"CRDS_PATH\"] = 'crds_cache'\n", + "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds.stsci.edu\"\n", + "import crds\n", + "\n", + "# for JWST calibration pipeline\n", + "import jwst\n", + "from jwst import datamodels\n", + "from jwst.pipeline import Detector1Pipeline, Image2Pipeline, Image3Pipeline\n", + "from jwst.associations import asn_from_list, mkpool\n", + "from jwst.associations.lib.rules_level3_base import DMS_Level3_Base\n", + "# For visualizing images\n", + "from jdaviz import Imviz\n", + "\n", + "# Astropy routines for visualizing detected sources:\n", + "from astropy.table import Table\n", + "from astropy.coordinates import SkyCoord\n", + "\n", + "# To confirm which version of the pipeline you're running:\n", + "print(f\"jwst pipeline version: {jwst.__version__}\")" + ] + }, + { + "cell_type": "markdown", + "id": "459d06b6-4f6a-402c-85a8-d06715a3d9e5", + "metadata": {}, + "source": [ + "## Define program parameters\n", + "\n", + "These will be used for generating the assocation file for Stage 3" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8d595f1e-2590-4f01-bb92-13bb43a22b3c", + "metadata": {}, + "outputs": [], + "source": [ + "# APT program ID number:\n", + "pid = '01475'\n", + "\n", + "data_dir = '1475_f150w/'" + ] + }, + { + "cell_type": "markdown", + "id": "881f784e-8df2-4012-9927-7aa19cdd8f67", + "metadata": {}, + "source": [ + "***\n", + "# 2. Examining uncalibrated data products" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ee9487b-a642-482f-a645-c8f8c2f8fb1b", + "metadata": {}, + "outputs": [], + "source": [ + "uncal_files = sorted(glob.glob(os.path.join(data_dir,'*_uncal.fits')))" + ] + }, + { + "cell_type": "markdown", + "id": "dca6413e-6a91-40c1-9bfa-0ebf78e6431b", + "metadata": {}, + "source": [ + "## Look at the first file to determine exposure parameters and practice using JWST datamodels" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f767a7fa-50cb-4411-8aef-a32b7fde8917", + "metadata": {}, + "outputs": [], + "source": [ + "# print file name\n", + "print(uncal_files[0])\n", + "\n", + "# Open file as JWST datamodel\n", + "examine = datamodels.open(uncal_files[0])\n", + "\n", + "# Print out exposure info\n", + "print(\"Instrument: \" + examine.meta.instrument.name)\n", + "print(\"Filter: \" + examine.meta.instrument.filter)\n", + "print(\"Pupil: \" + examine.meta.instrument.pupil)\n", + "print(\"Number of integrations: {}\".format(examine.meta.exposure.nints))\n", + "print(\"Number of groups: {}\".format(examine.meta.exposure.ngroups))\n", + "print(\"Readout pattern: \" + examine.meta.exposure.readpatt)\n", + "print(\"Dither position number: {}\".format(examine.meta.dither.position_number))\n", + "print(\"\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "0469a648-9bb1-40f9-9f8b-fb3a29bc5c7f", + "metadata": {}, + "source": [ + "From the above, we confirm that the data file is for the NIRISS instrument using the `F150W` filter in the [Pupil Wheel](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-instrumentation/niriss-pupil-and-filter-wheels) crossed with the `CLEAR` filter in the Filter Wheel. This observation uses the [`NIS` readout pattern](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-instrumentation/niriss-detector-overview/niriss-detector-readout-patterns), 16 groups per integration, and 1 integration per exposure. This data file is the 1st dither position in this exposure sequence. For more information about how JWST exposures are defined by up-the-ramp sampling, see the [Understanding Exposure Times JDox article](https://jwst-docs.stsci.edu/understanding-exposure-times).\n", + "\n", + "This metadata will be the same for all exposures in this observation other than the dither position number." + ] + }, + { + "cell_type": "markdown", + "id": "46be5421-7fc8-4d96-b2cd-2484f294e082", + "metadata": {}, + "source": [ + "## Display uncalibrated image\n", + "\n", + "We can visualize an uncalibrated dataset that will show detector artifacts that will be removed when calibrating the data through the `DETECTOR1` stage of the pipeline. Uncalibrated data files thus are 4D: nintegrations x ngroups x nrows x ncols. Here, we are visualizing the full detector (i.e., all columns and rows) and the 1st group.\n", + "\n", + "We are using the [Imviz tool](https://jdaviz.readthedocs.io/en/latest/imviz/index.html) within the `jdaviz` package to visualize the NIRISS image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1db43de4-896e-4eb4-8e92-aecf67eed5be", + "metadata": {}, + "outputs": [], + "source": [ + "# Create an Imviz instance and set up default viewer\n", + "imviz_uncal = Imviz()\n", + "viewer_uncal = imviz_uncal.default_viewer\n", + "\n", + "# Read in the science array for our visualization dataset:\n", + "uncal_science = examine.data\n", + "\n", + "# Load the dataset into Imviz\n", + "imviz_uncal.load_data(uncal_science[0,0,:,:])\n", + "\n", + "# Visualize the dataset:\n", + "imviz_uncal.show()" + ] + }, + { + "cell_type": "markdown", + "id": "b2d33e40-64b1-4d24-9a72-af586ccddf0c", + "metadata": {}, + "source": [ + "Adjust settings for the viewer:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "00479015-d39a-493e-9c0f-300823901bc1", + "metadata": {}, + "outputs": [], + "source": [ + "viewer_uncal.stretch = 'sqrt'\n", + "viewer_uncal.set_colormap('Viridis')\n", + "viewer_uncal.cuts = '99.5%'" + ] + }, + { + "cell_type": "markdown", + "id": "768c0cdb-72a3-4b2f-90f8-161f52b6d954", + "metadata": {}, + "source": [ + "***\n", + "# 3. Stage 1 Processing \n", + "Run the datasets through the [Detector1 stage of the pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_detector1) to apply detector level calibrations and create a countrate data product where slopes are fitted to the integration ramps. These `*_rate.fits` products are 2D (nrows x ncols), averaged over all integrations. 3D countrate data products (`*_rateints.fits`) are also created (nintegrations x nrows x ncols) which have the fitted ramp slopes for each integration.\n", + "\n", + "The [pipeline documentation](https://jwst-pipeline.readthedocs.io/en/latest/jwst/user_documentation/running_pipeline_python.html) discusses how to run the pipeline in the Python Interface, including how to configure pipeline steps and override reference files. By returning the results of the pipeline to a variable, the pipeline returns a datamodel. Note that the [`pipeline.call()` method is preferred over the `pipeline.run()` method](https://jwst-pipeline.readthedocs.io/en/latest/jwst/user_documentation/running_pipeline_python.html#advanced-use-pipeline-run-vs-pipeline-call).\n", + "\n", + "By default, [all steps in the Detector1 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_detector1.html) are run for NIRISS except: the `ipc` correction step and the `gain_scale` step. Note that while the [`persistence` step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/persistence/description.html) is set to run by default, this step does not automatically correct the science data for persistence. The `persistence` step creates a `*_trapsfilled.fits` file which is a model that records the number of traps filled at each pixel at the end of an exposure. This file would be used as an input to the `persistence` step, via the `input_trapsfilled` argument, to correct a science exposure for persistence. Since persistence is not well calibrated for NIRISS, we do not perform a persistence correction and thus turn off this step to speed up calibration and to not create files that will not be used in the subsequent analysis. This step can be turned off when running the pipeline in Python by doing:\n", + "```\n", + "rate_result = Detector1Pipeline.call(uncal,\n", + " steps={'persistence': {'skip': True}})\n", + "```\n", + "\n", + "The [charge_migration step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/charge_migration/index.html#charge-migration-step) is particularly important for NIRISS images to mitigate apparent flux loss in resampled images due to the spilling of charge from a central pixel into its neighboring pixels (see [Goudfrooij et al. 2023](https://ui.adsabs.harvard.edu/abs/2023arXiv231116301G/abstract) for details). Charge migration occurs when the accumulated charge in a central pixel exceeds a certain signal limit, which is ~25,000 ADU. This step is turned on by default for NIRISS imaging, Wide Field Slitless Spectroscopy [(WFSS)](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-observing-modes/niriss-wide-field-slitless-spectroscopy), and Aperture Masking Interferometry [(AMI)](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-observing-modes/niriss-aperture-masking-interferometry) modes when using CRDS contexts of `jwst_1159.pmap` or later. Different signal limits for each filter are provided by the [pars-chargemigrationstep parameter files](https://jwst-crds.stsci.edu). Users can specify a different signal limit by running this step with the `signal_threshold` flag and entering another signal limit in units of ADU.\n", + "\n", + "As of CRDS context `jwst_1155.pmap` and later, the [`jump` step](https://jwst-pipeline.readthedocs.io/en/latest/api/jwst.jump.JumpStep.html) of the `DETECTOR1` stage of the pipeline will remove residuals associated with [snowballs](https://jwst-docs.stsci.edu/data-artifacts-and-features/snowballs-and-shower-artifacts) for NIRISS imaging, WFSS, and AMI modes. The default parameters for this correction, where `expand_large_events` set to `True` turns on the snowball halo removal algorithm, are specified in the `pars-jumpstep` parameter reference files. Users may wish to alter parameters to optimize removal of snowball residuals. Available parameters are discussed in the [Detection and Flagging of Showers and Snowballs in JWST Technical Report (Regan 2023)](https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-documents/_documents/JWST-STScI-008545.pdf)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0f8f107b-5f80-4674-bc34-2d88791e4409", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Define directory to save output from detector1\n", + "det1_dir = 'detector1/'\n", + "# Create directory if it does not exist\n", + "if not os.path.isdir(det1_dir):\n", + " os.mkdir(det1_dir)\n", + "\n", + "# Run Detector1 stage of pipeline, specifying:\n", + "# output directory to save *_rate.fits files\n", + "# save_results flag set to True so the rate files are saved\n", + "# skipping the persistence step\n", + " \n", + "for uncal in uncal_files:\n", + " rate_result = Detector1Pipeline.call(uncal,\n", + " output_dir=det1_dir,\n", + " save_results=True,\n", + " steps={'persistence': {'skip': True}})" + ] + }, + { + "cell_type": "markdown", + "id": "7fc5520a-6097-482b-b3e0-a6bf94cf7af3", + "metadata": {}, + "source": [ + "## Identify `*_rate.fits` files" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "26c5dca4-2390-4919-a7e6-9fae755aeda6", + "metadata": {}, + "outputs": [], + "source": [ + "rate_files = sorted(glob.glob(os.path.join(det1_dir,'*_rate.fits')))" + ] + }, + { + "cell_type": "markdown", + "id": "f5845ca6-5280-489b-a1da-c2267857d034", + "metadata": {}, + "source": [ + "## Verify which pipeline steps were run and which calibration reference files were applied\n", + "\n", + "The header contains information about which calibration steps were completed and skipped and which reference files were used to process the data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1c122766-a545-4042-93e8-f68c18f62fdf", + "metadata": {}, + "outputs": [], + "source": [ + "# Read in file as datamodel\n", + "rate_f = datamodels.open(rate_files[0])\n", + "\n", + "rate_f.meta.cal_step.instance" + ] + }, + { + "cell_type": "markdown", + "id": "da6698a5-d1ce-4043-80a1-8b9ff35108b3", + "metadata": {}, + "source": [ + "Check which reference files were used to calibrate the dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5f1430b7-9106-4e13-b668-394cdcc53420", + "metadata": {}, + "outputs": [], + "source": [ + "rate_f.meta.ref_file.instance" + ] + }, + { + "cell_type": "markdown", + "id": "3ff813f4-f9d6-4765-b882-48f954b8969d", + "metadata": {}, + "source": [ + "## Display rate image\n", + "\n", + "Visualize a countrate image, using the dataset from the first dither position as an example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "176e27f5-f59f-4a6c-960b-4d688288eae3", + "metadata": {}, + "outputs": [], + "source": [ + "# Create an Imviz instance and set up default viewer\n", + "imviz_rate = Imviz()\n", + "viewer_rate = imviz_rate.default_viewer\n", + "\n", + "# Read in the science array for our visualization dataset:\n", + "rate_science = rate_f.data\n", + "\n", + "# Load the dataset into Imviz\n", + "imviz_rate.load_data(rate_science)\n", + "\n", + "# Visualize the dataset:\n", + "imviz_rate.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb6f488c-eaef-4b96-9284-07f0e1d13f6b", + "metadata": {}, + "outputs": [], + "source": [ + "viewer_rate.stretch = 'sqrt'\n", + "viewer_rate.set_colormap('Viridis')\n", + "viewer_rate.cuts = '95%'" + ] + }, + { + "cell_type": "markdown", + "id": "350808ee-9579-4cb5-8f02-a74904c91449", + "metadata": {}, + "source": [ + "***\n", + "# 3. Stage 2 Processing \n", + "\n", + "In the [Image2 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image2.html), calibrated unrectified data products are created (`*_cal.fits` or `*_calints.fits` files, depending on whether the input files are `*_rate.fits` or `*_rateints.fits`). \n", + "\n", + "In this pipeline processing stage, the [world coordinate system (WCS)](https://jwst-pipeline.readthedocs.io/en/latest/jwst/assign_wcs/index.html#assign-wcs-step) is assigned, the data are [flat fielded](https://jwst-pipeline.readthedocs.io/en/latest/jwst/flatfield/index.html#flatfield-step), and a [photometric calibration](https://jwst-pipeline.readthedocs.io/en/latest/jwst/photom/index.html#photom-step) is applied to convert from units of countrate (ADU/s) to surface brightness (MJy/sr). \n", + "\n", + "By default, the [background subtraction step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/background_step/index.html#background-step) and the [resampling step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) are turned off for NIRISS at this stage of the pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "81d916e5-dd5d-472d-9aff-b2b4c86decf8", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Define directory to save output from Image2\n", + "image2_dir = 'image2/'\n", + "# Create directory if it does not exist\n", + "if not os.path.isdir(image2_dir):\n", + " os.mkdir(image2_dir)\n", + "\n", + "# Run Image2 stage of pipeline, specifying:\n", + "# output directory to save *_cal.fits files\n", + "# save_results flag set to True so the rate files are saved\n", + "\n", + "for rate in rate_files:\n", + " cal_result = Image2Pipeline.call(rate,\n", + " output_dir=image2_dir,\n", + " save_results=True)" + ] + }, + { + "cell_type": "markdown", + "id": "75b26730-40b9-4ea7-a6ad-47727f00e401", + "metadata": {}, + "source": [ + "## Identify `*_cal.fits` files" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e80bb4d0-858f-4779-b927-c32332f7a670", + "metadata": {}, + "outputs": [], + "source": [ + "cal_files = sorted(glob.glob(os.path.join(image2_dir,'*_cal.fits')))" + ] + }, + { + "cell_type": "markdown", + "id": "da57b7d0-95ef-4227-b730-6f54a3eaaa16", + "metadata": {}, + "source": [ + "## Verify which pipeline steps were run" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "27390bbd-ae2d-49e7-977a-59d3765c0946", + "metadata": {}, + "outputs": [], + "source": [ + "cal_f = datamodels.open(cal_files[0])\n", + "\n", + "cal_f.meta.cal_step.instance" + ] + }, + { + "cell_type": "markdown", + "id": "0bd56084-72bb-4491-acd3-c8faeabca331", + "metadata": {}, + "source": [ + "Check which reference files were used to calibrate the dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a8aced8-773b-4f8b-86f4-2c62d3b3bcc2", + "metadata": {}, + "outputs": [], + "source": [ + "cal_f.meta.ref_file.instance" + ] + }, + { + "cell_type": "markdown", + "id": "19b9d7e4-536c-4661-ac9a-d21e7daf1389", + "metadata": {}, + "source": [ + "## Display cal image\n", + "\n", + "Visualize a calibrated image, using the dataset from the first dither position as an example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15455eda-74b5-453b-b5c9-e21b48b73446", + "metadata": {}, + "outputs": [], + "source": [ + "# Create an Imviz instance and set up default viewer\n", + "imviz_cal = Imviz()\n", + "viewer_cal = imviz_cal.default_viewer\n", + "\n", + "# Read in the science array for our visualization dataset:\n", + "cal_science = cal_f.data\n", + "\n", + "# Load the dataset into Imviz\n", + "imviz_cal.load_data(cal_science)\n", + "\n", + "# Visualize the dataset:\n", + "imviz_cal.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2b89947f-c9c6-4eee-a4e1-0f880d28bf31", + "metadata": {}, + "outputs": [], + "source": [ + "viewer_cal.stretch = 'sqrt'\n", + "viewer_cal.set_colormap('Viridis')\n", + "viewer_cal.cuts = '95%'" + ] + }, + { + "cell_type": "markdown", + "id": "7cfbb244-7af9-4cbf-90d9-08598378c16a", + "metadata": {}, + "source": [ + "***\n", + "# 3. Stage 3 Processing \n", + "\n", + "In the [Image3 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image3.html), the individual `*_cal.fits` files for each of the dither positions are combined to one single distortion corrected image. First, an [Association](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/overview.html) needs to be created to inform the pipeline that these individual exposures are linked together. \n", + "\n", + "By default, the `IMAGE3` stage of the pipeline performs the following steps on NIRISS data: \n", + "- [tweakreg](https://jwst-pipeline.readthedocs.io/en/latest/jwst/tweakreg/index.html#tweakreg-step) - creates source catalogs of pointlike sources for each input image. The source catalog for each input image is compared to each other to derive coordinate transforms to align the images relative to each other.\n", + " - As of CRDS context `jwst_1156.pmap` and later, the `pars-tweakreg` parameter reference file for NIRISS performs an absolute astrometric correction to GAIA data release 3 by default (i.e., the `abs_refcat` parameter is set to `GAIADR3`). Though this default correction generally improves results compared with not doing this alignment, it can sometimes result in poor performance in crowded or sparse fields, so users are encouraged to check astrometric accuracy and revisit this step if necessary.\n", + " - As of pipeline version 1.12.5, the default source finding algorithm is `DAOStarFind` which can result in up to 0.5 pix uncertainties in the centroids for undersampled PSFs, like the NIRISS PSFs at short wavelengths [(Goudfrooij 2022)](https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-documents/_documents/JWST-STScI-008116.pdf). There are plans to update the default algorithm to `IRAFStarFind` in future pipeline versions.\n", + "- [skymatch](https://jwst-pipeline.readthedocs.io/en/latest/jwst/skymatch/index.html#skymatch-step) - measures the background level from the sky to use as input into the subsequent `outlier detection` and `resample` steps.\n", + "- [outlier detection](https://jwst-pipeline.readthedocs.io/en/latest/jwst/outlier_detection/index.html#outlier-detection-step) - flags any remaining cosmic rays, bad pixels, or other artifacts not already flagged during the `DETECTOR1` stage of the pipeline, using all input images to create a median image so that outliers in individual images can be identified.\n", + "- [resample](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) - resamples each input image based on its WCS and distortion information and creates a single undistorted image.\n", + "- [source catalog](https://jwst-pipeline.readthedocs.io/en/latest/jwst/source_catalog/index.html#source-catalog-step) - creates a catalog of detected sources along with measured photometries and morphologies (i.e., point-like vs extended). Useful for quicklooks, but optimization is likely needed for specific science cases.\n" + ] + }, + { + "cell_type": "markdown", + "id": "db5cb6f5-44a1-4f28-95dc-b23a97451465", + "metadata": {}, + "source": [ + "## Create Association File\n", + "\n", + "An association file lists the exposures to calibrated together in `Stage 3` of the pipeline. Note that an association file is available for download from MAST, with a filename of `*_asn.json`. Here we show how to create an association file to point to the data products created when processing data through the pipeline. Note that the output products will have a rootname that is specified by the `product_name` in the association file. For this tutorial, the rootname of the output products will be `image3_association`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b31f80f-8d7b-45ae-8537-15d0208637df", + "metadata": {}, + "outputs": [], + "source": [ + "# Create a Level 3 Association\n", + "associations = asn_from_list.asn_from_list(cal_files, \n", + " rule=DMS_Level3_Base, \n", + " product_name='image3_association')\n", + "\n", + "associations.data['asn_type'] = 'image3'\n", + "associations.data['program'] = pid\n", + "\n", + "# Format association as .json file\n", + "asn_filename, serialized = associations.dump(format=\"json\")\n", + "\n", + "# Write out association file\n", + "with open(asn_filename,\"w\") as fd:\n", + " fd.write(serialized)" + ] + }, + { + "cell_type": "markdown", + "id": "629cad5b-e5a7-473e-8583-ad12cda55fa4", + "metadata": {}, + "source": [ + "## Run Image3 stage of the pipeline\n", + "\n", + "Given the grouped exposures in the association file, the `Image3` stage of the pipeline will produce:\n", + "- a `*_cr.fits` file produced by the `outlier_detection` step, where the `DQ` array marks the pixels flagged as outliers.\n", + "- a final combined, rectified image with name `*_i2d.fits`,\n", + "- a source catalog with name `*_cat.ecsv`,\n", + "- a segmentation map file (`*_segm.fits`) which has integer values at the pixel locations where a source is detected where the pixel values match the source ID number in the catalog." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e0cebfd8-a650-41a3-8ff0-a8fd43e079d6", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# Define directory to save output from Image2\n", + "image3_dir = 'image3/'\n", + "# Create directory if it does not exist\n", + "if not os.path.isdir(image3_dir):\n", + " os.mkdir(image3_dir)\n", + "\n", + "# Run Stage 3\n", + "i2d_result = Image3Pipeline.call(asn_filename, \n", + " output_dir=image3_dir,\n", + " save_results=True)" + ] + }, + { + "cell_type": "markdown", + "id": "4bcb6c40-fbb8-4226-b620-64b0c02c8ceb", + "metadata": {}, + "source": [ + "## Verify which pipeline steps were run" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c754530a-d4e2-493b-bf8b-bf8d1ad08770", + "metadata": {}, + "outputs": [], + "source": [ + "# Identify *_i2d file and open as datamodel\n", + "i2d = glob.glob(os.path.join(image3_dir,\"*_i2d.fits\"))[0]\n", + "i2d_f = datamodels.open(i2d)\n", + "\n", + "i2d_f.meta.cal_step.instance" + ] + }, + { + "cell_type": "markdown", + "id": "c0a53657-ad44-4430-a7b0-070c6be8fa2c", + "metadata": {}, + "source": [ + "Check which reference files were used to calibrate the dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2ee94ea-dd2b-43d4-ad7d-ea427810df0d", + "metadata": {}, + "outputs": [], + "source": [ + "i2d_f.meta.ref_file.instance" + ] + }, + { + "cell_type": "markdown", + "id": "8d44a0de-774a-4d63-8de9-80a9c6aa24b2", + "metadata": {}, + "source": [ + "## Display combined image\n", + "\n", + "Visualize the drizzle-combined image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c6f1ed63-0bed-412b-ab04-54caa7d61bbd", + "metadata": {}, + "outputs": [], + "source": [ + "# Create an Imviz instance and set up default viewer\n", + "imviz_i2d = Imviz()\n", + "viewer_i2d = imviz_i2d.default_viewer\n", + "\n", + "# Read in the science array for our visualization dataset:\n", + "i2d_science = i2d_f.data\n", + "\n", + "# Load the dataset into Imviz\n", + "imviz_i2d.load_data(i2d_science)\n", + "\n", + "# Visualize the dataset:\n", + "imviz_i2d.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8d22a117-e4cf-4d2f-9fec-f52545c8aa5a", + "metadata": {}, + "outputs": [], + "source": [ + "viewer_i2d.stretch = 'sqrt'\n", + "viewer_i2d.set_colormap('Viridis')\n", + "viewer_i2d.cuts = '95%'" + ] + }, + { + "cell_type": "markdown", + "id": "018733eb-07ec-44a4-a5a7-d6d2f27d193a", + "metadata": {}, + "source": [ + "# Visualize Detected Sources\n", + "Using the source catalog created by the `IMAGE3` stage of the pipeline, mark the detected sources, using different markers for point sources and extended sources. The source catalog is saved in `image3/image3_association_cat.ecsv` file. We will need to read in the `i2d` file again to make sure the world coordinate system (WCS) info is read in." + ] + }, + { + "cell_type": "markdown", + "id": "2ad6ee55-f940-4be3-96c9-114bc83eb784", + "metadata": {}, + "source": [ + "## Read in catalog file and identify point/extended sources" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "32235fdc-4ec2-42e6-89e7-b9f62824e179", + "metadata": {}, + "outputs": [], + "source": [ + "catalog_file = glob.glob(os.path.join(image3_dir,\"*_cat.ecsv\"))[0]\n", + "catalog = Table.read(catalog_file)\n", + "\n", + "# To identify point/extended sources, use the 'is_extended' column in the source catalog\n", + "pt_src, = np.where(catalog['is_extended'] == False)\n", + "ext_src, = np.where(catalog['is_extended'] == True)\n", + "\n", + "# Define coordinates of point and extended sources\n", + "pt_coord = Table({'coord':[SkyCoord(ra=catalog['sky_centroid'][pt_src].ra,\n", + " dec=catalog['sky_centroid'][pt_src].dec)]})\n", + "ext_coord = Table({'coord':[SkyCoord(ra=catalog['sky_centroid'][ext_src].ra,\n", + " dec=catalog['sky_centroid'][ext_src].dec)]})" + ] + }, + { + "cell_type": "markdown", + "id": "be9e9069-375d-431b-bbfb-e84e6cefc2d4", + "metadata": {}, + "source": [ + "## Mark the extended and point sources on the image\n", + "\n", + "Display combined image:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "302bc3ce-358d-4ea8-a533-37b9a9a2152c", + "metadata": {}, + "outputs": [], + "source": [ + "# Read in i2d file to Imviz\n", + "imviz_cat = Imviz()\n", + "viewer_cat = imviz_cat.default_viewer\n", + "imviz_cat.load_data(i2d)\n", + "\n", + "# Adjust settings for viewer\n", + "viewer_cat.stretch = 'sqrt'\n", + "viewer_cat.set_colormap('Viridis')\n", + "viewer_cat.cuts = '95%'\n", + "\n", + "\n", + "imviz_cat.show()" + ] + }, + { + "cell_type": "markdown", + "id": "bb9dc081-7a24-442c-bf86-ed2faa6cf342", + "metadata": {}, + "source": [ + "Point sources will be marked by small pink circles and extended sources will be marked by larger white circles." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fcb8f619-3807-44d1-a8d9-fbf41abb5522", + "metadata": {}, + "outputs": [], + "source": [ + "# Add marker for point sources:\n", + "viewer_cat.marker = {'color': 'pink', 'markersize':50, 'fill':False}\n", + "\n", + "viewer_cat.add_markers(pt_coord, use_skycoord=True, marker_name='point_sources')\n", + "\n", + "# Add marker for extended sources:\n", + "viewer_cat.marker = {'color': 'white', 'markersize':100, 'fill': False}\n", + "\n", + "viewer_cat.add_markers(ext_coord, use_skycoord=True, marker_name='extended_sources')\n" + ] + }, + { + "cell_type": "markdown", + "id": "1c1d9abd-b2d4-4c5d-a5ed-a1ef98e136fd", + "metadata": {}, + "source": [ + "From zoooming in on different portions of the image, we can see that some saturated point sources are erroneously labeled as point sources, some extended and saturated objects are not detected, parts of diffraction spikes are sometimes detected as \"extended sources\", and in some cases, the detected source centroid is offset from the center of a source. It is recommended for users to optimize source detection for their science goals using either their own tools or by updating parameters to the `source_catalog` step of the pipeline." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aad427db-1935-4261-b46e-346f1230522f", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From f8a3d820c5493c537432de1cd8cf79f90338186c Mon Sep 17 00:00:00 2001 From: Stephanie LaMassa Date: Tue, 23 Jan 2024 16:37:34 -0500 Subject: [PATCH 2/8] adding NIRISS imaging pipeline tutorial notebook --- .../niriss-imaging-tutorial.ipynb | 48 +++++++++---------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index a8c705d61..b2b255d02 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -21,7 +21,7 @@ "id": "dc6b2aba-072e-4179-a856-29d547f277d6", "metadata": {}, "source": [ - "# 1. Introduction\n", + "## 1. Introduction\n", "In this notebook, we will process a [NIRISS imaging dataset](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-observing-modes/niriss-imaging) through the [JWST calibration pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview). The example dataset is from [Program ID](https://www.stsci.edu/jwst/science-execution/program-information) 1475 (PI: Boyer, CoI: Volk) which is a sky flat calibration program. NIRCam is used as the primary instrument with NIRISS as a [coordinated parallel instrument](https://jwst-docs.stsci.edu/methods-and-roadmaps/jwst-parallel-observations/jwst-coordinated-parallels-roadmap). The NIRISS imaging dataset uses a 17-step dither pattern.\n", "\n", "For illustrative purposes, we focus on data taken through the NIRISS [F150W filter](https://jwst-docs.stsci.edu/jwst-near-infrared-imager-and-slitless-spectrograph/niriss-instrumentation/niriss-filters) and start with uncalibrated data products. The files are named\n", @@ -35,7 +35,7 @@ "id": "026a649d-a0d2-4e3f-ab6d-cf8c26e6ce1d", "metadata": {}, "source": [ - "## Install pipeline and dependencies" + "### Install pipeline and dependencies" ] }, { @@ -50,7 +50,7 @@ "pip install jdaviz --upgrade\n", "```\n", "\n", - "## Imports" + "### Imports" ] }, { @@ -90,7 +90,7 @@ "id": "459d06b6-4f6a-402c-85a8-d06715a3d9e5", "metadata": {}, "source": [ - "## Define program parameters\n", + "### Define program parameters\n", "\n", "These will be used for generating the assocation file for Stage 3" ] @@ -114,7 +114,7 @@ "metadata": {}, "source": [ "***\n", - "# 2. Examining uncalibrated data products" + "## 2. Examining uncalibrated data products" ] }, { @@ -132,7 +132,7 @@ "id": "dca6413e-6a91-40c1-9bfa-0ebf78e6431b", "metadata": {}, "source": [ - "## Look at the first file to determine exposure parameters and practice using JWST datamodels" + "### Look at the first file to determine exposure parameters and practice using JWST datamodels" ] }, { @@ -174,7 +174,7 @@ "id": "46be5421-7fc8-4d96-b2cd-2484f294e082", "metadata": {}, "source": [ - "## Display uncalibrated image\n", + "### Display uncalibrated image\n", "\n", "We can visualize an uncalibrated dataset that will show detector artifacts that will be removed when calibrating the data through the `DETECTOR1` stage of the pipeline. Uncalibrated data files thus are 4D: nintegrations x ngroups x nrows x ncols. Here, we are visualizing the full detector (i.e., all columns and rows) and the 1st group.\n", "\n", @@ -228,7 +228,7 @@ "metadata": {}, "source": [ "***\n", - "# 3. Stage 1 Processing \n", + "## 3. Stage 1 Processing \n", "Run the datasets through the [Detector1 stage of the pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_detector1) to apply detector level calibrations and create a countrate data product where slopes are fitted to the integration ramps. These `*_rate.fits` products are 2D (nrows x ncols), averaged over all integrations. 3D countrate data products (`*_rateints.fits`) are also created (nintegrations x nrows x ncols) which have the fitted ramp slopes for each integration.\n", "\n", "The [pipeline documentation](https://jwst-pipeline.readthedocs.io/en/latest/jwst/user_documentation/running_pipeline_python.html) discusses how to run the pipeline in the Python Interface, including how to configure pipeline steps and override reference files. By returning the results of the pipeline to a variable, the pipeline returns a datamodel. Note that the [`pipeline.call()` method is preferred over the `pipeline.run()` method](https://jwst-pipeline.readthedocs.io/en/latest/jwst/user_documentation/running_pipeline_python.html#advanced-use-pipeline-run-vs-pipeline-call).\n", @@ -276,7 +276,7 @@ "id": "7fc5520a-6097-482b-b3e0-a6bf94cf7af3", "metadata": {}, "source": [ - "## Identify `*_rate.fits` files" + "### Identify `*_rate.fits` files" ] }, { @@ -294,7 +294,7 @@ "id": "f5845ca6-5280-489b-a1da-c2267857d034", "metadata": {}, "source": [ - "## Verify which pipeline steps were run and which calibration reference files were applied\n", + "### Verify which pipeline steps were run and which calibration reference files were applied\n", "\n", "The header contains information about which calibration steps were completed and skipped and which reference files were used to process the data." ] @@ -335,7 +335,7 @@ "id": "3ff813f4-f9d6-4765-b882-48f954b8969d", "metadata": {}, "source": [ - "## Display rate image\n", + "### Display rate image\n", "\n", "Visualize a countrate image, using the dataset from the first dither position as an example." ] @@ -379,7 +379,7 @@ "metadata": {}, "source": [ "***\n", - "# 3. Stage 2 Processing \n", + "## 3. Stage 2 Processing \n", "\n", "In the [Image2 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image2.html), calibrated unrectified data products are created (`*_cal.fits` or `*_calints.fits` files, depending on whether the input files are `*_rate.fits` or `*_rateints.fits`). \n", "\n", @@ -418,7 +418,7 @@ "id": "75b26730-40b9-4ea7-a6ad-47727f00e401", "metadata": {}, "source": [ - "## Identify `*_cal.fits` files" + "### Identify `*_cal.fits` files" ] }, { @@ -436,7 +436,7 @@ "id": "da57b7d0-95ef-4227-b730-6f54a3eaaa16", "metadata": {}, "source": [ - "## Verify which pipeline steps were run" + "### Verify which pipeline steps were run" ] }, { @@ -474,7 +474,7 @@ "id": "19b9d7e4-536c-4661-ac9a-d21e7daf1389", "metadata": {}, "source": [ - "## Display cal image\n", + "### Display cal image\n", "\n", "Visualize a calibrated image, using the dataset from the first dither position as an example." ] @@ -518,7 +518,7 @@ "metadata": {}, "source": [ "***\n", - "# 3. Stage 3 Processing \n", + "## 3. Stage 3 Processing \n", "\n", "In the [Image3 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image3.html), the individual `*_cal.fits` files for each of the dither positions are combined to one single distortion corrected image. First, an [Association](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/overview.html) needs to be created to inform the pipeline that these individual exposures are linked together. \n", "\n", @@ -537,7 +537,7 @@ "id": "db5cb6f5-44a1-4f28-95dc-b23a97451465", "metadata": {}, "source": [ - "## Create Association File\n", + "### Create Association File\n", "\n", "An association file lists the exposures to calibrated together in `Stage 3` of the pipeline. Note that an association file is available for download from MAST, with a filename of `*_asn.json`. Here we show how to create an association file to point to the data products created when processing data through the pipeline. Note that the output products will have a rootname that is specified by the `product_name` in the association file. For this tutorial, the rootname of the output products will be `image3_association`." ] @@ -570,7 +570,7 @@ "id": "629cad5b-e5a7-473e-8583-ad12cda55fa4", "metadata": {}, "source": [ - "## Run Image3 stage of the pipeline\n", + "### Run Image3 stage of the pipeline\n", "\n", "Given the grouped exposures in the association file, the `Image3` stage of the pipeline will produce:\n", "- a `*_cr.fits` file produced by the `outlier_detection` step, where the `DQ` array marks the pixels flagged as outliers.\n", @@ -605,7 +605,7 @@ "id": "4bcb6c40-fbb8-4226-b620-64b0c02c8ceb", "metadata": {}, "source": [ - "## Verify which pipeline steps were run" + "### Verify which pipeline steps were run" ] }, { @@ -645,7 +645,7 @@ "id": "8d44a0de-774a-4d63-8de9-80a9c6aa24b2", "metadata": {}, "source": [ - "## Display combined image\n", + "### Display combined image\n", "\n", "Visualize the drizzle-combined image." ] @@ -688,7 +688,7 @@ "id": "018733eb-07ec-44a4-a5a7-d6d2f27d193a", "metadata": {}, "source": [ - "# Visualize Detected Sources\n", + "## Visualize Detected Sources\n", "Using the source catalog created by the `IMAGE3` stage of the pipeline, mark the detected sources, using different markers for point sources and extended sources. The source catalog is saved in `image3/image3_association_cat.ecsv` file. We will need to read in the `i2d` file again to make sure the world coordinate system (WCS) info is read in." ] }, @@ -697,7 +697,7 @@ "id": "2ad6ee55-f940-4be3-96c9-114bc83eb784", "metadata": {}, "source": [ - "## Read in catalog file and identify point/extended sources" + "### Read in catalog file and identify point/extended sources" ] }, { @@ -726,7 +726,7 @@ "id": "be9e9069-375d-431b-bbfb-e84e6cefc2d4", "metadata": {}, "source": [ - "## Mark the extended and point sources on the image\n", + "### Mark the extended and point sources on the image\n", "\n", "Display combined image:" ] @@ -811,7 +811,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.13" + "version": "3.10.6" } }, "nbformat": 4, From 34ed3a544c4bb34a91282a2275cc0099ba21c10f Mon Sep 17 00:00:00 2001 From: Gilbert Green Date: Tue, 6 Feb 2024 15:34:27 -0500 Subject: [PATCH 3/8] added requirements.txt file, updating notebook styling to adhere to PEP8 guidelines --- .../niriss-imaging-tutorial.ipynb | 53 ++++++++----------- notebooks/niriss_imaging/requirements.txt | 5 ++ 2 files changed, 28 insertions(+), 30 deletions(-) create mode 100644 notebooks/niriss_imaging/requirements.txt diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index b2b255d02..d30d4c51d 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -60,20 +60,17 @@ "metadata": {}, "outputs": [], "source": [ - "import os, glob\n", + "import os\n", + "import glob\n", "import numpy as np\n", "\n", - "# Configure CRDS\n", - "os.environ[\"CRDS_PATH\"] = 'crds_cache'\n", - "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds.stsci.edu\"\n", - "import crds\n", - "\n", "# for JWST calibration pipeline\n", "import jwst\n", "from jwst import datamodels\n", "from jwst.pipeline import Detector1Pipeline, Image2Pipeline, Image3Pipeline\n", - "from jwst.associations import asn_from_list, mkpool\n", + "from jwst.associations import asn_from_list\n", "from jwst.associations.lib.rules_level3_base import DMS_Level3_Base\n", + "\n", "# For visualizing images\n", "from jdaviz import Imviz\n", "\n", @@ -81,6 +78,10 @@ "from astropy.table import Table\n", "from astropy.coordinates import SkyCoord\n", "\n", + "# Configure CRDS\n", + "os.environ[\"CRDS_PATH\"] = 'crds_cache'\n", + "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds.stsci.edu\"\n", + "\n", "# To confirm which version of the pipeline you're running:\n", "print(f\"jwst pipeline version: {jwst.__version__}\")" ] @@ -124,7 +125,7 @@ "metadata": {}, "outputs": [], "source": [ - "uncal_files = sorted(glob.glob(os.path.join(data_dir,'*_uncal.fits')))" + "uncal_files = sorted(glob.glob(os.path.join(data_dir, '*_uncal.fits')))" ] }, { @@ -196,7 +197,7 @@ "uncal_science = examine.data\n", "\n", "# Load the dataset into Imviz\n", - "imviz_uncal.load_data(uncal_science[0,0,:,:])\n", + "imviz_uncal.load_data(uncal_science[0, 0, :, :])\n", "\n", "# Visualize the dataset:\n", "imviz_uncal.show()" @@ -286,7 +287,7 @@ "metadata": {}, "outputs": [], "source": [ - "rate_files = sorted(glob.glob(os.path.join(det1_dir,'*_rate.fits')))" + "rate_files = sorted(glob.glob(os.path.join(det1_dir, '*_rate.fits')))" ] }, { @@ -428,7 +429,7 @@ "metadata": {}, "outputs": [], "source": [ - "cal_files = sorted(glob.glob(os.path.join(image2_dir,'*_cal.fits')))" + "cal_files = sorted(glob.glob(os.path.join(image2_dir, '*_cal.fits')))" ] }, { @@ -561,7 +562,7 @@ "asn_filename, serialized = associations.dump(format=\"json\")\n", "\n", "# Write out association file\n", - "with open(asn_filename,\"w\") as fd:\n", + "with open(asn_filename, \"w\") as fd:\n", " fd.write(serialized)" ] }, @@ -616,7 +617,7 @@ "outputs": [], "source": [ "# Identify *_i2d file and open as datamodel\n", - "i2d = glob.glob(os.path.join(image3_dir,\"*_i2d.fits\"))[0]\n", + "i2d = glob.glob(os.path.join(image3_dir, \"*_i2d.fits\"))[0]\n", "i2d_f = datamodels.open(i2d)\n", "\n", "i2d_f.meta.cal_step.instance" @@ -707,18 +708,18 @@ "metadata": {}, "outputs": [], "source": [ - "catalog_file = glob.glob(os.path.join(image3_dir,\"*_cat.ecsv\"))[0]\n", + "catalog_file = glob.glob(os.path.join(image3_dir, \"*_cat.ecsv\"))[0]\n", "catalog = Table.read(catalog_file)\n", "\n", "# To identify point/extended sources, use the 'is_extended' column in the source catalog\n", - "pt_src, = np.where(catalog['is_extended'] == False)\n", - "ext_src, = np.where(catalog['is_extended'] == True)\n", + "pt_src, = np.where(~catalog['is_extended'])\n", + "ext_src, = np.where(catalog['is_extended'])\n", "\n", "# Define coordinates of point and extended sources\n", - "pt_coord = Table({'coord':[SkyCoord(ra=catalog['sky_centroid'][pt_src].ra,\n", - " dec=catalog['sky_centroid'][pt_src].dec)]})\n", - "ext_coord = Table({'coord':[SkyCoord(ra=catalog['sky_centroid'][ext_src].ra,\n", - " dec=catalog['sky_centroid'][ext_src].dec)]})" + "pt_coord = Table({'coord': [SkyCoord(ra=catalog['sky_centroid'][pt_src].ra,\n", + " dec=catalog['sky_centroid'][pt_src].dec)]})\n", + "ext_coord = Table({'coord': [SkyCoord(ra=catalog['sky_centroid'][ext_src].ra,\n", + " dec=catalog['sky_centroid'][ext_src].dec)]})" ] }, { @@ -768,12 +769,12 @@ "outputs": [], "source": [ "# Add marker for point sources:\n", - "viewer_cat.marker = {'color': 'pink', 'markersize':50, 'fill':False}\n", + "viewer_cat.marker = {'color': 'pink', 'markersize': 50, 'fill': False}\n", "\n", "viewer_cat.add_markers(pt_coord, use_skycoord=True, marker_name='point_sources')\n", "\n", "# Add marker for extended sources:\n", - "viewer_cat.marker = {'color': 'white', 'markersize':100, 'fill': False}\n", + "viewer_cat.marker = {'color': 'white', 'markersize': 100, 'fill': False}\n", "\n", "viewer_cat.add_markers(ext_coord, use_skycoord=True, marker_name='extended_sources')\n" ] @@ -785,14 +786,6 @@ "source": [ "From zoooming in on different portions of the image, we can see that some saturated point sources are erroneously labeled as point sources, some extended and saturated objects are not detected, parts of diffraction spikes are sometimes detected as \"extended sources\", and in some cases, the detected source centroid is offset from the center of a source. It is recommended for users to optimize source detection for their science goals using either their own tools or by updating parameters to the `source_catalog` step of the pipeline." ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "aad427db-1935-4261-b46e-346f1230522f", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/notebooks/niriss_imaging/requirements.txt b/notebooks/niriss_imaging/requirements.txt new file mode 100644 index 000000000..95f344571 --- /dev/null +++ b/notebooks/niriss_imaging/requirements.txt @@ -0,0 +1,5 @@ +numpy +crds +jwst>=1.2.3 +jdaviz +astropy>=4.3.1 From 72cd815e375b758b6cfd6e68e29788c608954a89 Mon Sep 17 00:00:00 2001 From: Gilbert Green Date: Wed, 7 Feb 2024 15:19:34 -0500 Subject: [PATCH 4/8] removing a line of whitespace for PEP8 --- notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index d30d4c51d..8f94929d1 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -776,7 +776,7 @@ "# Add marker for extended sources:\n", "viewer_cat.marker = {'color': 'white', 'markersize': 100, 'fill': False}\n", "\n", - "viewer_cat.add_markers(ext_coord, use_skycoord=True, marker_name='extended_sources')\n" + "viewer_cat.add_markers(ext_coord, use_skycoord=True, marker_name='extended_sources')" ] }, { From 43ec5174fd2a259454240d259d5461b87a3de9e0 Mon Sep 17 00:00:00 2001 From: Gilbert Green Date: Thu, 8 Feb 2024 12:58:35 -0500 Subject: [PATCH 5/8] manual rebase of main --- .../niriss-imaging-tutorial.ipynb | 33 +++++++++++++------ 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index 8f94929d1..7446434a3 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -5,7 +5,7 @@ "id": "0393e357-9d9d-4516-b28d-4d335fad33a0", "metadata": {}, "source": [ - "# Baseline: NIRISS Imaging Tutorial Notebook\n", + "# Advanced: NIRISS Imaging Tutorial Notebook\n", "\n", "## Table of Contents\n", "1. [Introduction](#intro)\n", @@ -13,7 +13,9 @@ "3. [Stage 1 Processing](#det1)\n", "4. [Stage 2 Processing](#image2)\n", "5. [Stage 3 Processing](#image3)\n", - "6. [Visualize Detected Sources](#detections)" + "6. [Visualize Detected Sources](#detections)\n", + "\n", + "Date published: January 24, 2024" ] }, { @@ -62,6 +64,8 @@ "source": [ "import os\n", "import glob\n", + "import zipfile\n", + "import urllib.request\n", "import numpy as np\n", "\n", "# for JWST calibration pipeline\n", @@ -91,9 +95,7 @@ "id": "459d06b6-4f6a-402c-85a8-d06715a3d9e5", "metadata": {}, "source": [ - "### Define program parameters\n", - "\n", - "These will be used for generating the assocation file for Stage 3" + "### Download uncalibrated data products" ] }, { @@ -106,7 +108,19 @@ "# APT program ID number:\n", "pid = '01475'\n", "\n", - "data_dir = '1475_f150w/'" + "# Set up directory to download uncalibrated data files:\n", + "data_dir = '1475_f150w/'\n", + "# Create directory if it does not exist\n", + "if not os.path.isdir(data_dir):\n", + " os.mkdir(data_dir)\n", + "\n", + "# Download uncalibrated data from Box into current directory:\n", + "boxlink = 'https://data.science.stsci.edu/redirect/JWST/jwst-data_analysis_tools/niriss_imaging/1475_f150w.zip'\n", + "boxfile = os.path.join(data_dir, '1475_f150w.zip')\n", + "urllib.request.urlretrieve(boxlink, boxfile)\n", + "\n", + "zf = zipfile.ZipFile(boxfile, 'r')\n", + "zf.extractall()" ] }, { @@ -386,7 +400,7 @@ "\n", "In this pipeline processing stage, the [world coordinate system (WCS)](https://jwst-pipeline.readthedocs.io/en/latest/jwst/assign_wcs/index.html#assign-wcs-step) is assigned, the data are [flat fielded](https://jwst-pipeline.readthedocs.io/en/latest/jwst/flatfield/index.html#flatfield-step), and a [photometric calibration](https://jwst-pipeline.readthedocs.io/en/latest/jwst/photom/index.html#photom-step) is applied to convert from units of countrate (ADU/s) to surface brightness (MJy/sr). \n", "\n", - "By default, the [background subtraction step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/background_step/index.html#background-step) and the [resampling step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) are turned off for NIRISS at this stage of the pipeline." + "By default, the [background subtraction step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/background_step/index.html#background-step) and the [resampling step](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) are turned off for NIRISS at this stage of the pipeline. The background subtraction is turned off since there is no background template for the imaging mode and the local background is removed during the background correction for photometric measurements around individual sources. The resampling step occurs during the `Image3` stage by default. While the resampling step can be turned on during the `Image2` stage to, e.g., generate a source catalog for each image, the data quality from the `Image3` stage will be better since the bad pixels, which adversely affect both the centroids and photometry in individual images, will be mostly removed." ] }, { @@ -523,14 +537,14 @@ "\n", "In the [Image3 stage of the pipeline](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image3.html), the individual `*_cal.fits` files for each of the dither positions are combined to one single distortion corrected image. First, an [Association](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/overview.html) needs to be created to inform the pipeline that these individual exposures are linked together. \n", "\n", - "By default, the `IMAGE3` stage of the pipeline performs the following steps on NIRISS data: \n", + "By default, the `Image3` stage of the pipeline performs the following steps on NIRISS data: \n", "- [tweakreg](https://jwst-pipeline.readthedocs.io/en/latest/jwst/tweakreg/index.html#tweakreg-step) - creates source catalogs of pointlike sources for each input image. The source catalog for each input image is compared to each other to derive coordinate transforms to align the images relative to each other.\n", " - As of CRDS context `jwst_1156.pmap` and later, the `pars-tweakreg` parameter reference file for NIRISS performs an absolute astrometric correction to GAIA data release 3 by default (i.e., the `abs_refcat` parameter is set to `GAIADR3`). Though this default correction generally improves results compared with not doing this alignment, it can sometimes result in poor performance in crowded or sparse fields, so users are encouraged to check astrometric accuracy and revisit this step if necessary.\n", " - As of pipeline version 1.12.5, the default source finding algorithm is `DAOStarFind` which can result in up to 0.5 pix uncertainties in the centroids for undersampled PSFs, like the NIRISS PSFs at short wavelengths [(Goudfrooij 2022)](https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-documents/_documents/JWST-STScI-008116.pdf). There are plans to update the default algorithm to `IRAFStarFind` in future pipeline versions.\n", "- [skymatch](https://jwst-pipeline.readthedocs.io/en/latest/jwst/skymatch/index.html#skymatch-step) - measures the background level from the sky to use as input into the subsequent `outlier detection` and `resample` steps.\n", "- [outlier detection](https://jwst-pipeline.readthedocs.io/en/latest/jwst/outlier_detection/index.html#outlier-detection-step) - flags any remaining cosmic rays, bad pixels, or other artifacts not already flagged during the `DETECTOR1` stage of the pipeline, using all input images to create a median image so that outliers in individual images can be identified.\n", "- [resample](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) - resamples each input image based on its WCS and distortion information and creates a single undistorted image.\n", - "- [source catalog](https://jwst-pipeline.readthedocs.io/en/latest/jwst/source_catalog/index.html#source-catalog-step) - creates a catalog of detected sources along with measured photometries and morphologies (i.e., point-like vs extended). Useful for quicklooks, but optimization is likely needed for specific science cases.\n" + "- [source catalog](https://jwst-pipeline.readthedocs.io/en/latest/jwst/source_catalog/index.html#source-catalog-step) - creates a catalog of detected sources along with measured photometries and morphologies (i.e., point-like vs extended). Useful for quicklooks, but optimization is likely needed for specific science cases, which is an on-going investigation for the NIRISS team. Users may wish to experiment with changing the `snr_threshold` and `deblend` options. Modifications to the following parameters will not significantly improve data quality and it is advised to keep them at their default values: `aperture_ee1`, `aperture_ee2`, `aperture_ee3`, `ci1_star_threshold`, `ci2_star_threshold`.\n" ] }, { @@ -749,7 +763,6 @@ "viewer_cat.set_colormap('Viridis')\n", "viewer_cat.cuts = '95%'\n", "\n", - "\n", "imviz_cat.show()" ] }, From d9062f2547477680a4a6fdb32acf2002442bee03 Mon Sep 17 00:00:00 2001 From: gibsongreen Date: Thu, 15 Feb 2024 10:04:19 -0500 Subject: [PATCH 6/8] updated requirements file and rearrange CRDS and jwst pipeline --- .../niriss_imaging/niriss-imaging-tutorial.ipynb | 16 ++++++++-------- notebooks/niriss_imaging/requirements.txt | 2 ++ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index 7446434a3..2cd703dc7 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -68,13 +68,6 @@ "import urllib.request\n", "import numpy as np\n", "\n", - "# for JWST calibration pipeline\n", - "import jwst\n", - "from jwst import datamodels\n", - "from jwst.pipeline import Detector1Pipeline, Image2Pipeline, Image3Pipeline\n", - "from jwst.associations import asn_from_list\n", - "from jwst.associations.lib.rules_level3_base import DMS_Level3_Base\n", - "\n", "# For visualizing images\n", "from jdaviz import Imviz\n", "\n", @@ -86,6 +79,13 @@ "os.environ[\"CRDS_PATH\"] = 'crds_cache'\n", "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds.stsci.edu\"\n", "\n", + "# for JWST calibration pipeline\n", + "import jwst\n", + "from jwst import datamodels\n", + "from jwst.pipeline import Detector1Pipeline, Image2Pipeline, Image3Pipeline\n", + "from jwst.associations import asn_from_list\n", + "from jwst.associations.lib.rules_level3_base import DMS_Level3_Base\n", + "\n", "# To confirm which version of the pipeline you're running:\n", "print(f\"jwst pipeline version: {jwst.__version__}\")" ] @@ -822,4 +822,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/notebooks/niriss_imaging/requirements.txt b/notebooks/niriss_imaging/requirements.txt index 95f344571..efa42a15c 100644 --- a/notebooks/niriss_imaging/requirements.txt +++ b/notebooks/niriss_imaging/requirements.txt @@ -3,3 +3,5 @@ crds jwst>=1.2.3 jdaviz astropy>=4.3.1 +zipfile +urllib \ No newline at end of file From 4c218a3383988e28eb6e1c9928bc203e802642ef Mon Sep 17 00:00:00 2001 From: gibsongreen Date: Thu, 15 Feb 2024 10:11:53 -0500 Subject: [PATCH 7/8] removing urllib and zipfile from requirements --- notebooks/niriss_imaging/requirements.txt | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/notebooks/niriss_imaging/requirements.txt b/notebooks/niriss_imaging/requirements.txt index efa42a15c..c6fb7dc62 100644 --- a/notebooks/niriss_imaging/requirements.txt +++ b/notebooks/niriss_imaging/requirements.txt @@ -2,6 +2,4 @@ numpy crds jwst>=1.2.3 jdaviz -astropy>=4.3.1 -zipfile -urllib \ No newline at end of file +astropy>=4.3.1 \ No newline at end of file From e15b147f2efbf180ee10d8f607e2da8083bd53c2 Mon Sep 17 00:00:00 2001 From: gibsongreen Date: Tue, 12 Mar 2024 16:58:56 -0400 Subject: [PATCH 8/8] text changes to notebook --- .../niriss_imaging/niriss-imaging-tutorial.ipynb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb index 2cd703dc7..7e75ffe19 100755 --- a/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb +++ b/notebooks/niriss_imaging/niriss-imaging-tutorial.ipynb @@ -45,11 +45,11 @@ "id": "9c5b415a-f0f4-4c48-a038-d319e825ee4f", "metadata": {}, "source": [ - "Make sure the [latest version of the pipeline is installed](https://jwst-pipeline.readthedocs.io/en/stable/index.html) and that the [Calibration Reference Data System (CRDS)](https://jwst-crds.stsci.edu/static/users_guide/index.html) is configured so the most up-to-date calibration files are used to process the dataset. This notebook was created with pipeline version `1.12.3`. \n", - "\n", - "This notebook uses the [`jdaviz` package](https://jdaviz.readthedocs.io/en/latest/index.html) to visualize the datasets. To [install `jdaviz`](https://jdaviz.readthedocs.io/en/latest/installation.html) in your current environment:\n", + "To make sure that the pipeline version is compatabile with the steps discussed below and the required dependencies and packages are installed, you can create a fresh conda environment and install the provided `requirements.txt` file:\n", "```\n", - "pip install jdaviz --upgrade\n", + "conda create -n niriss_imaging_pipeline python=3.11\n", + "conda activate niriss_imaging_pipeline\n", + "pip install -r requirements.txt\n", "```\n", "\n", "### Imports" @@ -540,7 +540,7 @@ "By default, the `Image3` stage of the pipeline performs the following steps on NIRISS data: \n", "- [tweakreg](https://jwst-pipeline.readthedocs.io/en/latest/jwst/tweakreg/index.html#tweakreg-step) - creates source catalogs of pointlike sources for each input image. The source catalog for each input image is compared to each other to derive coordinate transforms to align the images relative to each other.\n", " - As of CRDS context `jwst_1156.pmap` and later, the `pars-tweakreg` parameter reference file for NIRISS performs an absolute astrometric correction to GAIA data release 3 by default (i.e., the `abs_refcat` parameter is set to `GAIADR3`). Though this default correction generally improves results compared with not doing this alignment, it can sometimes result in poor performance in crowded or sparse fields, so users are encouraged to check astrometric accuracy and revisit this step if necessary.\n", - " - As of pipeline version 1.12.5, the default source finding algorithm is `DAOStarFind` which can result in up to 0.5 pix uncertainties in the centroids for undersampled PSFs, like the NIRISS PSFs at short wavelengths [(Goudfrooij 2022)](https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-documents/_documents/JWST-STScI-008116.pdf). There are plans to update the default algorithm to `IRAFStarFind` in future pipeline versions.\n", + " - As of pipeline version 1.12.5, the default source finding algorithm is `DAOStarFinder` which can result in up to 0.5 pix uncertainties in the centroids for undersampled PSFs, like the NIRISS PSFs at short wavelengths [(Goudfrooij 2022)](https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-documents/_documents/JWST-STScI-008116.pdf). There are plans to update the default algorithm to `IRAFStarFinder` in future pipeline versions.\n", "- [skymatch](https://jwst-pipeline.readthedocs.io/en/latest/jwst/skymatch/index.html#skymatch-step) - measures the background level from the sky to use as input into the subsequent `outlier detection` and `resample` steps.\n", "- [outlier detection](https://jwst-pipeline.readthedocs.io/en/latest/jwst/outlier_detection/index.html#outlier-detection-step) - flags any remaining cosmic rays, bad pixels, or other artifacts not already flagged during the `DETECTOR1` stage of the pipeline, using all input images to create a median image so that outliers in individual images can be identified.\n", "- [resample](https://jwst-pipeline.readthedocs.io/en/latest/jwst/resample/index.html#resample-step) - resamples each input image based on its WCS and distortion information and creates a single undistorted image.\n", @@ -822,4 +822,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +}