[BUG]: Bug Description - pipeline chrashes during /lv3_step.py", line 460

[lib-j]

Describe the bug
Running pjpipe pipeline on MIRI data - using the https://github.com/PhangsTeam/pjpipe/tree/main/config/1558 files

Pipeline breaks during lv3_step.py", line 460 due to "AttributeError: 'ModelLibrary' object has no attribute '_models'"

To Reproduce
INFO:jwst.stpipe.core:Results used jwst version: 1.19.1
Traceback (most recent call last):
File "/Users/olivia.jones/Science/pjpipe_test/run_reprocessing.py", line 27, in
pjp.do_pipeline()
~~~~~~~~~~~~~~~^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/pipeline.py", line 669, in do_pipeline
step_result = lv3.do_step()
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/lv3/lv3_step.py", line 175, in do_step
success = self.run_step(
asn_file,
)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/lv3/lv3_step.py", line 460, in run_step
for i, model in enumerate(asn_file._models):
^^^^^^^^^^^^^^^^
AttributeError: 'ModelLibrary' object has no attribute '_models'

Expected behavior
Pipeline should complete.

Desktop (please complete the following information):

• OS: MacOS
• Version 15.5

Additional context
Clean conda environment so only PJpipe and associated dependencies are installed.

[thomaswilliamsastro]

I thought we'd fixed this a while back...just to check, are you using the pip version or installing from GitHub? If it's pip, try:

pip install git+https://github.com/PhangsTeam/pjpipe.git

and let me know if that works

[lib-j]

Thank you. I was using the pip version rather than the GitHub one. Unfortunately, I am now encountering a bug earlier in the process, in the get data stage. This error is not present in the pip-installed version, which fetched the data earlier.

For testing purposes, I am using:
https://github.com/PhangsTeam/pjpipe/blob/main/config/1558/config.toml

(pjpipe) olivia.jones@UKACTMAC12 pjpipe_test2 % python run_reprocessing.py
Traceback (most recent call last):
File "/Users/olivia.jones/Data/IZw18/MIRI_3533_Img/pjpipe_test/pjpipe_test2/run_reprocessing.py", line 27, in
pjp.do_pipeline()
~~~~~~~~~~~~~~~^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/pipeline.py", line 268, in do_pipeline
step_result = download.do_step()
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/download/download_step.py", line 206, in do_step
self.run_archive_query()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/pjpipe/download/download_step.py", line 252, in run_archive_query
self.obs_list = self.observations.query_object(self.target)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
return self.fn(obj, *args, **kwds)
~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/utils/process_asyncs.py", line 26, in newmethod
response = getattr(self, async_method_name)(*args, **kwargs)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
return self.fn(obj, *args, **kwds)
~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/mast/observations.py", line 271, in query_object_async
coordinates = utils.resolve_object(objectname)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/mast/utils.py", line 112, in resolve_object
response = _simple_request("{}/api/v0/invoke".format(conf.server), request_string)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/astroquery/mast/utils.py", line 88, in _simple_request
response.raise_for_status()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe/lib/python3.13/site-packages/requests/models.py", line 1026, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://mast.stsci.edu/api/v0/invoke?request=%7B%22service%22%3A%20%22Mast.Name.Lookup%22%2C%20%22params%22%3A%20%7B%22input%22%3A%20%22ngc6720%22%2C%20%22format%22%3A%20%22json%22%7D%7D

[thomaswilliamsastro]

Seems like MAST is down (boo!), https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html gives me a 404 right now. I think this is just wait until it's back up, unfortunately

[lib-j]

OK MAST is back up and running, but I am still experiencing issues with the GitHub version of PJPIPE. It crashes very early on in the run (see logs below).

I have verified this works with the pip installed version of pjpipe, so this issue only arose in https://github.com/PhangsTeam/pjpipe.git.

In both cases, I am seeing the same config files and code; the only thing I have changed is the conda environment. So I am not sure why one would run to the stage 3 crash, whilst the other breaks during the download stage, I think at "download_step.py", line 440, in get_products

The pjpipe_git environment has pip installed, the git version of pjpipe and https://github.com/ejwatkins-astro/spacepylot.git
The pjpipe_pip environment has pip installed, pip version of pjpipe and https://github.com/ejwatkins-astro/spacepylot.git

Error below:

(pjpipe_git) olivia.jones@UKACTMAC12 pjpipe_test2 % python run_reprocessing.py
100%|##########################################################################################################################################################################| 187/187 [03:16<00:00, 1.05s/it]
Traceback (most recent call last):
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 502, in check_fill_value
fill_value = np.asarray(fill_value, dtype=ndtype)
ValueError: invalid literal for int() with base 10: np.str('N/A')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 3267, in view
output = ndarray.view(self, dtype)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 3138, in array_finalize
self._fill_value = _check_fill_value(self._fill_value, self.dtype)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 508, in _check_fill_value
raise TypeError(err_msg % (fill_value, ndtype)) from e
TypeError: Cannot convert fill_value N/A to dtype int64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 502, in check_fill_value
fill_value = np.asarray(fill_value, dtype=ndtype)
ValueError: invalid literal for int() with base 10: np.str('N/A')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/olivia.jones/Data/IZw18/MIRI_3533_Img/pjpipe_test/pjpipe_test2/run_reprocessing.py", line 27, in
pjp.do_pipeline()
~~~~~~~~~~~~~~~^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/pjpipe/pipeline.py", line 268, in do_pipeline
step_result = download.do_step()
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/pjpipe/download/download_step.py", line 213, in do_step
success = self.run_download()
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/pjpipe/download/download_step.py", line 366, in run_download
products = self.get_products(self.obs_list)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/pjpipe/download/download_step.py", line 440, in get_products
mask = np.char.find(products["dataURI"], "_gs-") == -1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/astropy/table/column.py", line 342, in _compare
result = getattr(super(Column, self), op)(other)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 4288, in eq
return self._comparison(other, operator.eq)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 4242, in _comparison
sdata = self.data
^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/astropy/table/column.py", line 1702, in data
out = self.view(np.ma.MaskedArray)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 3272, in view
output = ndarray.view(self, dtype)
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 3138, in array_finalize
self._fill_value = _check_fill_value(self._fill_value, self.dtype)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/olivia.jones/opt/miniconda3/envs/pjpipe_git/lib/python3.13/site-packages/numpy/ma/core.py", line 508, in _check_fill_value
raise TypeError(err_msg % (fill_value, ndtype)) from e
TypeError: Cannot convert fill_value N/A to dtype int64

[thomaswilliamsastro]

@lib-j Sorry for the slow reponse, but I'm on this today! Ironically MAST was also down this morning so progress has been hindered, but I have a fix. Just want to test nothing else has broken since. Should be done in the next day or so

[thomaswilliamsastro]

@lib-j This should now be fixed in #92. Let me know if it's not, else we can close this issue. Use the GH version:

pip install git+https://github.com/PhangsTeam/pjpipe.git