Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miscellaneous updates to file paths and S3 workaround #84

Merged
merged 12 commits into from
Jan 9, 2025
Merged

Miscellaneous updates to file paths and S3 workaround #84

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
938f505
Update simulated product info
tddesjardins Jan 9, 2025
44d7be2
Typo fix
tddesjardins Jan 9, 2025
9330abf
Update data access for S3 fix
tddesjardins Jan 9, 2025
43ebeaa
Update photometry for S3 fix
tddesjardins Jan 9, 2025
ef070ea
Update viz for S3 fix
tddesjardins Jan 9, 2025
53a093c
Update elp for S3 fix
tddesjardins Jan 9, 2025
e4e5d16
Fix typo in spec notebook
tddesjardins Jan 9, 2025
3a0aa03
Add explanatory text to data discovery about S3 bug workaround
tddesjardins Jan 9, 2025
459cdcc
Fix file paths for spec notebook
tddesjardins Jan 9, 2025
a4a506e
Update shapes for S3 fix
tddesjardins Jan 9, 2025
cf9e40a
Update mosaic pipeline for S3 fix and add explanatory text
tddesjardins Jan 9, 2025
8a45122
Fix astrocut paths
tddesjardins Jan 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion content/notebooks/aperture_photometry/aperture_photometry.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
"- *matplotlib*\n",
"- *photutils* is an Astropy-affiliated package for photometry\n",
"- *roman_datamodels* opens and validates WFI data files\n",
"- *asdf* opens WFI data files\n",
"- *os* for checking if files exist\n",
"- *s3fs* streams data from Simple Storage Service (S3) buckets on Amazon Web Services (AWS)"
]
Expand All @@ -63,6 +64,7 @@
"outputs": [],
"source": [
"from astropy.table import Table\n",
"import asdf\n",
"import copy\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
Expand Down Expand Up @@ -141,7 +143,8 @@
" asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/AAS_WORKSHOP/'\n",
" asdf_file_uri = asdf_dir_uri + 'r0003201001001001004_0001_wfi01_f106_cal.asdf'\n",
" with fs.open(asdf_file_uri, 'rb') as f:\n",
" dm = rdm.open(f)\n",
" af = asdf.open(f)\n",
" dm = rdm.open(af)\n",
" image = dm.data.value.copy()\n",
" wcs = copy.deepcopy(dm.meta.wcs)"
]
Expand Down
9 changes: 6 additions & 3 deletions content/notebooks/data_discovery_and_access/data_discovery_and_access.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,9 @@
"\n",
"Below, we use `roman_datamodels` to read the ASDF file corresponding to the dense region as an example. To simplify the workflow we are providing a URI to the sample Roman data. During operations, the data would be referenced using the URI when perform queries through MAST or other data access methods that are currently under development.\n",
"\n",
"The file naming convention for Roman is quite elaborate as each includes all the relevant information about the observation. Please see the [Data Levels and Products](https://roman-docs.stsci.edu/data-handbook-home/wfi-data-format/data-levels-and-products) Roman documentation page for more information on the file naming conventions."
"The file naming convention for Roman is quite elaborate as each includes all the relevant information about the observation. Please see the [Data Levels and Products](https://roman-docs.stsci.edu/data-handbook-home/wfi-data-format/data-levels-and-products) Roman documentation page for more information on the file naming conventions.\n",
"\n",
"**Note:** In the cell below, we read in the ASDF file from the S3 bucket first using `asdf.open()` and then passing that `AsdfFile` object to the `roman_datamodels.open()` function. This is due to a recently discovered bug when reading files from S3 with `roman_datamodels`, and we expect this to be fixed soon. For now, please use `asdf.open()` when reading from S3 and then pass the object to `roman_datamodels`."
]
},
{
Expand All @@ -294,10 +296,11 @@
"metadata": {},
"outputs": [],
"source": [
"asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi11.asdf'\n",
"asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi11_f106_cal.asdf'\n",
"\n",
"with fs.open(asdf_file_uri, 'rb') as f:\n",
" dm = rdm.open(f)\n",
" af = asdf.open(f)\n",
" dm = rdm.open(af)\n",
" \n",
"print(dm.info())"
]
Expand Down
31 changes: 18 additions & 13 deletions content/notebooks/data_visualization/data_visualization.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@
"- *astropy.coordinates.SkyCoord* to create Python objects containing sky coordinate transforms\n",
"- *matplotlib.pyplot* for creating static image previews\n",
"- *jdaviz.Imviz* to examine Wide Field Instrument (WFI) images interactively\n",
"- *roman_datamodels* for opening Roman WFI ASDF files"
"- *roman_datamodels* for opening Roman WFI ASDF files\n",
"- *asdf* for opening Roman WFI ASDF files"
]
},
{
Expand All @@ -61,6 +62,7 @@
"\n",
"from jdaviz import Imviz\n",
"import roman_datamodels as rdm\n",
"import asdf\n",
"import s3fs"
]
},
Expand Down Expand Up @@ -110,9 +112,10 @@
"asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/'\n",
"fs = s3fs.S3FileSystem()\n",
"\n",
"asdf_file_uri = asdf_dir_uri + 'ROMANISIM/DENSE_REGION/R0.5_DP0.5_PA0/r0000101001001001001_01101_0001_WFI16_cal.asdf'\n",
"asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi01_f106_cal.asdf'\n",
"with fs.open(asdf_file_uri, 'rb') as f:\n",
" file = rdm.open(f)"
" af = asdf.open(f)\n",
" file = rdm.open(af)"
]
},
{
Expand Down Expand Up @@ -234,10 +237,11 @@
"asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/'\n",
"fs = s3fs.S3FileSystem()\n",
"\n",
"asdf_file_uri = asdf_dir_uri + 'ROMANISIM/DENSE_REGION/R0.5_DP0.5_PA0/r0000101001001001001_01101_0001_WFI16_cal.asdf'\n",
"asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi01_f106_cal.asdf'\n",
"with fs.open(asdf_file_uri, 'rb') as f:\n",
" file = rdm.open(f)\n",
" imviz.load_data(file, data_label='WFI16_POS1')\n",
" af = asdf.open(f)\n",
" file = rdm.open(af)\n",
" imviz.load_data(file, data_label='WFI01_POS1')\n",
" # imviz.load_data(file, ext='dq', data_label='WFI16 DQ')"
]
},
Expand Down Expand Up @@ -276,10 +280,11 @@
"metadata": {},
"outputs": [],
"source": [
"asdf_file_uri = asdf_dir_uri + 'ROMANISIM/DENSE_REGION/R0.47_DP0.51_PA0/r0000101001001001001_01101_0002_WFI16_cal.asdf'\n",
"asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0002_wfi01_f106_cal.asdf'\n",
"with fs.open(asdf_file_uri, 'rb') as f:\n",
" file = rdm.open(f)\n",
" imviz.load_data(file, data_label='WFI16_POS2')\n",
" af = asdf.open(f)\n",
" file = rdm.open(af)\n",
" imviz.load_data(file, data_label='WFI01_POS2')\n",
"\n",
"viewer = imviz.default_viewer\n",
"viewer.stretch_options\n",
Expand Down Expand Up @@ -386,7 +391,7 @@
"outputs": [],
"source": [
"# Read in catalog data from S3\n",
"cat_uri = asdf_dir_uri + 'ROMANISIM/CATALOGS_SCRIPTS/fullcat_101M_pared_ra0.50_dec0.50_WFI16.ecsv'\n",
"cat_uri = asdf_dir_uri + 'AAS_WORKSHOP/full_catalog.ecsv'\n",
"with fs.open(cat_uri, 'rb') as f:\n",
" tab = Table.read(f, format='ascii.ecsv')"
]
Expand All @@ -395,7 +400,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"There are a lot of sources in this file, but let's pare them down somewhat for display purposes. In this case, let's filter down to the brightest ($m_{AB}$ < 15) sources in the F158 filter. The flux columns in the table are in units of maggies, which may be converted to AB magnitudes as $m_{AB} = -2.5\\log_{10}(f)$ where $m_{AB}$ is the magnitude in AB mags and $f$ is the flux in maggies."
"There are a lot of sources in this file, but let's pare them down somewhat for display purposes. In this case, let's filter down to the brightest ($m_{AB}$ < 15) sources in the F106 filter. The flux columns in the table are in units of maggies, which may be converted to AB magnitudes as $m_{AB} = -2.5\\log_{10}(f)$ where $m_{AB}$ is the magnitude in AB mags and $f$ is the flux in maggies."
]
},
{
Expand All @@ -405,7 +410,7 @@
"outputs": [],
"source": [
"# Create SkyCoord objects\n",
"bright = np.where(-2.5 * np.log10(tab['F158']) < 15)\n",
"bright = np.where(-2.5 * np.log10(tab['F106']) < 15)\n",
"coords = Table({'coord': [SkyCoord(ra = tab['ra'][bright], dec = tab['dec'][bright], unit = 'deg')]})"
]
},
Expand Down Expand Up @@ -512,7 +517,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
"version": "3.11.11"
}
},
"nbformat": 4,
Expand Down
5 changes: 4 additions & 1 deletion content/notebooks/exposure_pipeline/exposure_pipeline.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
" Libraries used\n",
"- *romancal* for running the processing pipeline\n",
"- *roman_datamodels* for opening Roman WFI ASDF files\n",
"- *asdf* for opening Roman WFI ASDF files\n",
"- *os* for checking if files exist\n",
"- *s3fs* for streaming files from an S3 bucket"
]
Expand All @@ -53,6 +54,7 @@
"outputs": [],
"source": [
"import roman_datamodels as rdm\n",
"import asdf\n",
"import romancal\n",
"from romancal.pipeline import ExposurePipeline\n",
"import os\n",
Expand Down Expand Up @@ -203,7 +205,8 @@
"\n",
" asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi01_f106_cal.asdf'\n",
" with fs.open(asdf_file_uri, 'rb') as f:\n",
" dm = rdm.open(f)"
" af = asdf.open(f)\n",
" dm = rdm.open(af)"
]
},
{
Expand Down
400 changes: 54 additions & 346 deletions content/notebooks/grism_spectral_extraction/grism_spectral_extraction.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

17 changes: 10 additions & 7 deletions content/notebooks/measuring_galaxy_shapes/measuring_galaxy_shapes.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
"- *galsim* to measure source momens and generate Sérsic profiles\n",
"- *numpy* for array manipulation and mathematical operations\n",
"- *roman_datamodels* for opening WFI ASDF files\n",
"- *asdf* for opening WFI ASDF files\n",
"- *s3fs* for streaming simulated WFI images from an S3 bucket\n",
"- *scipy.optimize* to fit our data\n",
"- *synphot* for synthetic photometry\n",
Expand Down Expand Up @@ -74,6 +75,7 @@
"from galsim.roman import collecting_area\n",
"import numpy as np\n",
"import roman_datamodels as rdm\n",
"import asdf\n",
"import s3fs\n",
"from scipy.optimize import curve_fit\n",
"import synphot as syn\n",
Expand Down Expand Up @@ -141,10 +143,11 @@
"# Open an image from an S3 bucket\n",
"asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/'\n",
"fs = s3fs.S3FileSystem()\n",
"asdf_file_uri_l2 = asdf_dir_uri + 'ROMANISIM/GALAXIES/r0003201001001001004_01101_0001_WFI01_cal.asdf'\n",
"asdf_file_uri_l2 = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi01_f106_cal.asdf'\n",
"\n",
"with fs.open(asdf_file_uri_l2, 'rb') as f:\n",
" file = rdm.open(f).copy()"
" af = asdf.open(f)\n",
" file = rdm.open(af).copy()"
]
},
{
Expand Down Expand Up @@ -181,8 +184,8 @@
"outputs": [],
"source": [
"# Open the catalog used for the simulation as an astropy.table.Table\n",
"cat_uri = asdf_dir_uri + 'ROMANISIM/CATALOGS_SCRIPTS/galfield.ecsv'\n",
"with fs.open(cat_uri, 'r') as catalog_file_stream:\n",
"cat_uri = asdf_dir_uri + 'AAS_WORKSHOP/full_catalog.ecsv'\n",
"with fs.open(cat_uri, 'rb') as catalog_file_stream:\n",
" catalog = Table.read(cat_uri, format='ascii.ecsv').copy()"
]
},
Expand Down Expand Up @@ -940,9 +943,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "Roman Calibration latest (2024-03-25)",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "roman-cal"
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -954,7 +957,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
"version": "3.11.11"
}
},
"nbformat": 4,
Expand Down
24 changes: 17 additions & 7 deletions content/notebooks/mosaic_pipeline/mosaic_pipeline.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,17 @@
"outputs": [],
"source": [
"# Copy the files from the S3 bucket if they are not in local storage\n",
"#\n",
"# Uncomment the following lines if you do not have the output files \n",
"# from the tutorial \"Calibrating WFI Exposures with RomanCal\"\n",
"#\n",
"#"
"input_files = ['r0003201001001001004_0001_wfi11_f106_cal.asdf',\n",
" 'r0003201001001001004_0002_wfi11_f106_cal.asdf',\n",
" 'r0003201001001001004_0003_wfi11_f106_cal.asdf',\n",
" 'r0003201001001001004_0004_wfi11_f106_cal.asdf']\n",
"\n",
"for f in input_files:\n",
" if not os.path.exists(f):\n",
" fs = s3fs.S3FileSystem()\n",
" asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/AAS_WORKSHOP/'\n",
" asdf_file_uri = asdf_dir_uri + 'AAS_WORKSHOP/r0003201001001001004_0001_wfi11_f106_cal.asdf'\n",
" fs.get(asdf_file_uri, f'./{f}')"
]
},
{
Expand Down Expand Up @@ -240,9 +246,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# TO DO\n",
"An L3 file contains several extensions that may be of interest:\n",
"- data: The mosiac image in units of MegaJanskys per steradian\n",
"- context: An image showing the input images that contributed to each output pixel\n",
"- err: The error image\n",
"- weight: A weight map showing the relative coverage on the sky from the input images and how they contributed to the final mosaic\n",
"\n",
"Add info about L3 file contents (context, weight)."
"For more information on the how arrays are computed by the drizzle algorithm, we suggest consulting the [DrizzlePac Handbook](https://www.stsci.edu/files/live/sites/www/files/home/scientific-community/software/drizzlepac/_documents/drizzlepac-handbook.pdf). While the DrizzlePac Handbook is written for HST, many of the same terms and principles apply to Roman as the underlying algorithm is the same."
]
},
{
Expand Down
12 changes: 6 additions & 6 deletions content/notebooks/roman_cutouts/roman_cutouts.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@
"metadata": {},
"outputs": [],
"source": [
"asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/ROMANISIM'\n",
"asdf_dir_uri = 's3://roman-sci-test-data-prod-summer-beta-test/AAS_WORKSHOP/'\n",
"fs.ls(asdf_dir_uri)"
]
},
Expand All @@ -177,7 +177,7 @@
"outputs": [],
"source": [
"# URI to a level 2 image in S3 bucket\n",
"asdf_file_uri = asdf_dir_uri + '/DENSE_REGION/R0.5_DP0.5_PA0/r0000101001001001002_01101_0001_WFI18_cal.asdf'"
"asdf_file_uri = asdf_dir_uri + 'r0003201001001001004_0001_wfi01_f106_cal.asdf'"
]
},
{
Expand Down Expand Up @@ -300,7 +300,7 @@
"print('\\n')\n",
"\n",
"# get computed pixel coordinates from world coordinates with get_center_pixel\n",
"test_coord = SkyCoord('0.8 0.3', unit='deg') # test out different sky coordinates here!\n",
"test_coord = SkyCoord('270.8719 -0.16437', unit='deg') # test out different sky coordinates here!\n",
"print('Computed Closet Pixel Coordinate')\n",
"print('--------------------------------')\n",
"pc, ww = get_center_pixel(gwcs, test_coord.ra, test_coord.dec)\n",
Expand Down Expand Up @@ -787,9 +787,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "Roman Calibration latest (2024-03-25)",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "roman-cal"
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -801,7 +801,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
"version": "3.11.11"
}
},
"nbformat": 4,
Expand Down
Loading
Loading