Merge pull request #2550 from cta-observatory/prepare_v0.21.0

Prepare changelog for 0.21.0

Author: maxnoe

CHANGES.rst

@@ -1,3 +1,249 @@
+Ctapipe v0.21.0 (2024-04-25)
+============================
+
+
+API Changes
+-----------
+
+- ``reference_location`` is now a required argument for  ``SubarrayDescription``
+  [`#2402 <https://github.com/cta-observatory/ctapipe/pull/2402>`__]
+
+- ``CameraGeometry.position_to_pix_index`` will now return the minimum integer value for invalid
+  pixel coordinates instead of -1 due to the danger of using -1 as an index in python accessing
+  the last element of a data array for invalid pixels.
+  The function will now also no longer raise an error if the arguments are empty arrays and instead
+  just return an empty index array.
+  The function will also no longer log a warning in case of coordinates that do not match a camera pixel.
+  The function is very low-level and if not finding a pixel at the tested position warrants a warning or
+  is expected will depend on the calling code. [`#2397 <https://github.com/cta-observatory/ctapipe/pull/2397>`__]
+
+- Change the definition of the ``leakage_pixels_width_{1,2}`` image features
+  to give the ratio of pixels at the border to the pixels after cleaning
+  instead of to the total number of pixels of the camera. [`#2432 <https://github.com/cta-observatory/ctapipe/pull/2432>`__]
+
+- Change how the ``DataWriter`` writes pointing information.
+  Before, each unique pointing position was written in a table
+  with the event time as index column into ``dl1/monitoring/telescope/pointing``.
+
+  This has two issues: For observed data, each pointing will be unique
+  in horizontal coordinates due to tracking a fixed ICRS coordinate.
+  Resulting in a pointing position written for each event, although the
+  resolution of the monitoring is much lower.
+  For simulated events, the event time is the timestamp of the simulation
+  and pointing is fixed in ``AltAz``.
+  ``ctapipe`` was using the closest point in time for simulated events when
+  reading data back in, however, this is problematic in case of many
+  simulation runs processed in parallel.
+
+  We now store the first received pointing information
+  in the ``configuration/telescope/pointing`` table per obs id,
+  only for simulation events. [`#2438 <https://github.com/cta-observatory/ctapipe/pull/2438>`__]
+
+- Replace ``n_signal`` and ``n_background`` options in ``ctapipe-train-particle-classifier``
+  with ``n_events`` and ``signal_fraction``, where ``signal_fraction`` = n_signal / (n_signal + n_background). [`#2465 <https://github.com/cta-observatory/ctapipe/pull/2465>`__]
+
+- Move the ``TableLoader`` options from being traitlets to
+  each ``read_...`` method allowing to load different data with the
+  same TableLoader-Instance.
+
+  In addition the default values for the options have changed. [`#2482 <https://github.com/cta-observatory/ctapipe/pull/2482>`__]
+
+- Adding monitoring: MonitoringCameraContainer as keyword argument to
+  the ``ImageCleaner`` API so cleaning algorithms can now access
+  relevant information for methods that e.g. require monitoring information. [`#2511 <https://github.com/cta-observatory/ctapipe/pull/2511>`__]
+
+- Unified the options for DataWriter and the data level names:
+
+  +-------------------------+--------------------------+
+  | Old                     | New                      |
+  +=========================+==========================+
+  | ``write_raw_waveforms`` | ``write_r0_waveforms``   |
+  +-------------------------+--------------------------+
+  | ``write_waveforms``     | ``write_r1_waveforms``   |
+  +-------------------------+--------------------------+
+  | ``write_images``        | ``write_dl1_images``     |
+  +-------------------------+--------------------------+
+  | ``write_parameters``    | ``write_dl1_parameters`` |
+  +-------------------------+--------------------------+
+  | ``write_showers``       | ``write_dl2``            |
+  +-------------------------+--------------------------+
+
+  This changes requires that existing configuration files are updated
+  if they use these parameters [`#2520 <https://github.com/cta-observatory/ctapipe/pull/2520>`__]
+
+
+Bug Fixes
+---------
+
+- Ensure that ``SubarrayDescription.reference_location`` is always generated by
+  ```SimTelEventSource``, even if the metadata is missing. In that case, construct a
+  dummy location with the correct observatory height and latitude and longitude
+  equal to zero ("Null Island").
+
+- Fixed the definition of ``h_max``, which was both inconsistent between
+  `~ctapipe.reco.HillasReconstructor` and `~ctapipe.reco.HillasIntersection`
+  implementations, and was also incorrect since it was measured from the
+  observatory elevation rather than from sea level.
+
+  The value of ``h_max`` is now defined as the height above sea level of the
+  shower-max point (in meters), not the distance to that point. Therefore it is
+  not corrected for the zenith angle of the shower. This is consistent with the
+  options currently used for *CORSIKA*, where the *SLANT* option is set to false,
+  meaning heights are actual heights not distances from the impact point, and
+  ``x_max`` is a *depth*, not a *slant depth*. Note that this definition may be
+  inconsistent with other observatories where slant-depths are used, and also note
+  that the slant depth or distance to shower max are the more useful quantities
+  for shower physics. [`#2403 <https://github.com/cta-observatory/ctapipe/pull/2403>`__]
+
+- Add the example config for ctapipe-train-disp-reconstructor
+  to the list of configs generated by ctapipe-quickstart. [`#2414 <https://github.com/cta-observatory/ctapipe/pull/2414>`__]
+
+- Do not use a hidden attribute of ``SKLearnReconstructor`` in ``ctapipe-apply-models``. [`#2418 <https://github.com/cta-observatory/ctapipe/pull/2418>`__]
+
+- Add docstring for ``ctapipe-train-disp-reconstructor``. [`#2420 <https://github.com/cta-observatory/ctapipe/pull/2420>`__]
+
+- Remove warnings about missing R1 or DL0 data when using the CameraCalibrator.
+  These were previously emitted directly as python warnings and did not use the
+  component logging system, which they now do.
+  As we do not actually expect R1 to be present it was also moved down to
+  debug level. [`#2421 <https://github.com/cta-observatory/ctapipe/pull/2421>`__]
+
+- Check that the array pointing is given in horizontal coordinates
+  before training a ``DispReconstructor``. [`#2431 <https://github.com/cta-observatory/ctapipe/pull/2431>`__]
+
+- Fix additional, unwanted columns being written into disp prediction output. [`#2440 <https://github.com/cta-observatory/ctapipe/pull/2440>`__]
+
+- Properly transform pixel coordinates between ``CameraFrame``
+  and ``TelescopeFrame`` in ``MuonIntensityFitter`` taking.
+  Before, ``MuonIntensityFitter`` always used the equivalent focal
+  length for transformations, now it is using the focal length
+  attached to the ``CameraGeometry``, thus respecting the
+  ``focal_length_choice`` options of the event sources. [`#2464 <https://github.com/cta-observatory/ctapipe/pull/2464>`__]
+
+- Fix colored logging in case of custom log levels being defined. [`#2505 <https://github.com/cta-observatory/ctapipe/pull/2505>`__]
+
+- Fix a possible out-of-bounds array access in the FlashCamExtractor. [`#2544 <https://github.com/cta-observatory/ctapipe/pull/2544>`__]
+
+
+Data Model Changes
+------------------
+
+- Remove redundant ``is_valid`` field in ``DispContainer`` and rename the remaining field.
+  Use the same prefix for both containers filled by ``DispReconstructor``.
+
+  Fix default name of ``DispReconstructor`` target column.
+
+  Let ``HDF5EventSource`` load ``DispContainer``. [`#2443 <https://github.com/cta-observatory/ctapipe/pull/2443>`__]
+
+- Change R1- and DL0-waveforms datamodel shape from (n_pixels, n_samples)
+  to be always (n_channels, n_pixels, n_samples). ``HDF5EventSource`` was adjusted
+  accordingly to support also older datamodel versions.
+
+  Re-introduce also the possibility of running ``ImageExtractor``\s on data
+  consisting of multiple gain channels. [`#2529 <https://github.com/cta-observatory/ctapipe/pull/2529>`__]
+
+
+New Features
+------------
+
+- Large updates to the Image Pixel-wise fit for Atmospheric Cherenkov Telescopes reconstruction method (https://doi.org/10.48550/arXiv.1403.2993)
+
+  * ImPACT - General code clean up and optimisation. Now updated to work similarly to other reconstructors using the standardised interface, such that it can be used ctapipe-process. Significant improvements to tests too
+  * ImPACT_utilities - Created new file to hold general usage functions, numba used in some areas for speedup
+  * template_network_interpolator - Now works with templates with different zenith and azimuth angles
+  * unstructured_interpolator - Significant speed improvements
+  * pixel_likelihood - Constants added back to neg_log_likelihood_approx, these are quite important to obtaining a well normalised goodness of fit.
+  * hillas_intersection - Fixed bug in core position being incorrectly calculated, fixed tests too [`#2305 <https://github.com/cta-observatory/ctapipe/pull/2305>`__]
+
+- Allow passing the matplotlib axes to the ``SubarrayDescription.peek`` function,
+  fix warnings in case of layout engine being already defined. [`#2369 <https://github.com/cta-observatory/ctapipe/pull/2369>`__]
+
+- Add support for interpolating a monitoring pointing table
+  in ``TableLoader``. The corresponding table is not yet written by ``ctapipe``,
+  but can be written by external tools.
+  This is to enable analysis of real observations, where the pointing changes over time in
+  alt/az. [`#2409 <https://github.com/cta-observatory/ctapipe/pull/2409>`__]
+
+- Implement the overburden-to height a.s.l. transformation function in the atmosphere module
+  and test that round-trip returns original value. [`#2422 <https://github.com/cta-observatory/ctapipe/pull/2422>`__]
+
+- In case no configuration is found for a telescope in ``TelescopeParameter``,
+  it is now checked whether the telescope exists at all to provide a better
+  error message. [`#2429 <https://github.com/cta-observatory/ctapipe/pull/2429>`__]
+
+- Allow setting n_jobs on the command line for the
+  train_* and apply_models tools using a new ``n_jobs`` flag.
+  This temporarily overwrites any settings in the (model) config(s). [`#2430 <https://github.com/cta-observatory/ctapipe/pull/2430>`__]
+
+- Add support for using ``str`` and ``Path`` objects as input
+  to ``ctapipe.io.get_hdf5_datalevels``. [`#2451 <https://github.com/cta-observatory/ctapipe/pull/2451>`__]
+
+- The recommended citation for ctapipe has been updated to the ICRC 2023 proceeding,
+  please update. [`#2470 <https://github.com/cta-observatory/ctapipe/pull/2470>`__]
+
+- Support astropy 6.0. [`#2475 <https://github.com/cta-observatory/ctapipe/pull/2475>`__]
+
+- The ``DispReconstructor`` now computes a score for how certain the prediction of the disp sign is. [`#2479 <https://github.com/cta-observatory/ctapipe/pull/2479>`__]
+
+- Also load the new fixed pointing information in ``TableLoader``.
+
+  Add option ``keep_order`` to ``ctapipe.io.astropy_helpers.join_allow_empty``
+  that will keep the original order of rows when performing left or right joins. [`#2481 <https://github.com/cta-observatory/ctapipe/pull/2481>`__]
+
+- Add an ``AstroQuantity`` trait which can hold any ``astropy.units.Quantity``. [`#2524 <https://github.com/cta-observatory/ctapipe/pull/2524>`__]
+
+- Add function ``ctapipe.coordinates.get_point_on_shower_axis``
+  that computes a point on the shower axis in alt/az as seen
+  from a telescope. [`#2537 <https://github.com/cta-observatory/ctapipe/pull/2537>`__]
+
+- Update bokeh dependency to version 3.x. [`#2549 <https://github.com/cta-observatory/ctapipe/pull/2549>`__]
+
+
+Maintenance
+-----------
+
+- The CI system now reports to the CTA SonarQube instance for code quality tracking [`#2214 <https://github.com/cta-observatory/ctapipe/pull/2214>`__]
+
+- Updated some numpy calls to not use deprecated functions. [`#2406 <https://github.com/cta-observatory/ctapipe/pull/2406>`__]
+
+- The ``ctapipe`` source code repository now uses the ``src/``-based layout.
+  This fixes the editable installation of ctapipe. [`#2459 <https://github.com/cta-observatory/ctapipe/pull/2459>`__]
+
+- Fix headings in docs. Change occurrences of ``API Reference`` to ``Reference/API`` for consistency.
+  Change capitalization of some headings for consistency. [`#2474 <https://github.com/cta-observatory/ctapipe/pull/2474>`__]
+
+- The ``from_name`` methods of instrument description classes now raise a warning
+  that it is better to access instrument information via a ``SubarrayDescription``.
+
+  Also improve documentation in instrument module to explain when not to use the
+  various ``from_name()`` methods. These are provided for the case when no event
+  data is available, e.g. for unit testing or demos, but do not guarantee that the
+  returned instrument information corresponds to a particular set of event data. [`#2485 <https://github.com/cta-observatory/ctapipe/pull/2485>`__]
+
+- Support and test on python 3.12. [`#2486 <https://github.com/cta-observatory/ctapipe/pull/2486>`__]
+
+- Drop support for python 3.9. [`#2526 <https://github.com/cta-observatory/ctapipe/pull/2526>`__]
+
+
+Refactoring and Optimization
+----------------------------
+
+- Load data and apply event and column selection in chunks in ``ctapipe-train-*``
+  before merging afterwards.
+  This reduces memory usage. [`#2423 <https://github.com/cta-observatory/ctapipe/pull/2423>`__]
+
+- Make default ML config files more readable and add comments. [`#2455 <https://github.com/cta-observatory/ctapipe/pull/2455>`__]
+
+- Update and add missing docstrings related to the ML functionalities. [`#2456 <https://github.com/cta-observatory/ctapipe/pull/2456>`__]
+
+- Add ``true_impact_distance`` to the output of ``CrossValidator``. [`#2468 <https://github.com/cta-observatory/ctapipe/pull/2468>`__]
+
+- Add ``cache=True`` to some numba-compiled functions which were missing it. [`#2477 <https://github.com/cta-observatory/ctapipe/pull/2477>`__]
+
+- Write cross validation results for each model out immediately after validation to free up memory earlier. [`#2483 <https://github.com/cta-observatory/ctapipe/pull/2483>`__]
+
+- Compute deconvolution parameters in FlashCamExtractor only as needed. [`#2545 <https://github.com/cta-observatory/ctapipe/pull/2545>`__]
+
 ctapipe v0.20.0 (2023-09-11)
 ============================
 

docs/_static/switcher.json

@@ -10,35 +10,25 @@
     "url": "https://ctapipe.readthedocs.io/en/stable/"
   },
   {
-    "name": "v0.19.3",
-    "version": "v0.19.3",
-    "url": "https://ctapipe.readthedocs.io/en/v0.19.3/"
-  },
-  {
-    "name": "v0.19.2",
-    "version": "v0.19.2",
-    "url": "https://ctapipe.readthedocs.io/en/v0.19.2/"
+    "name": "v0.21.0",
+    "version": "v0.21.0",
+    "url": "https://ctapipe.readthedocs.io/en/v0.21.0/"
   },
   {
-    "name": "v0.19.1",
-    "version": "v0.19.1",
-    "url": "https://ctapipe.readthedocs.io/en/v0.19.1/"
+    "name": "v0.20.0",
+    "version": "v0.20.0",
+    "url": "https://ctapipe.readthedocs.io/en/v0.20.0/"
   },
   {
-    "name": "v0.19.0",
-    "version": "v0.19.0",
-    "url": "https://ctapipe.readthedocs.io/en/v0.19.0/"
+    "name": "v0.19.3",
+    "version": "v0.19.3",
+    "url": "https://ctapipe.readthedocs.io/en/v0.19.3/"
   },
   {
     "name": "v0.18.1",
     "version": "v0.18.1",
     "url": "https://ctapipe.readthedocs.io/en/v0.18.1/"
   },
-  {
-    "name": "v0.18.0",
-    "version": "v0.18.0",
-    "url": "https://ctapipe.readthedocs.io/en/v0.18.0/"
-  },
   {
     "name": "v0.17.0",
     "version": "v0.17.0",