v0.26.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

API Changes

• Changes in the environment variables set in the .env file.

  Added explicit setting of the simulation software branches:

  • CORSIKA: e.g., SIMTOOLS_CORSIKA_PATH=/workdir/simulation_software/corsika7
  • sim_telarray: e.g., SIMTOOLS_SIMTEL_PATH=/workdir/simulation_software/sim_telarray

  Added explicit setting of the interaction models:

  • high-energy interaction model: e.g., SIMTOOLS_CORSIKA_HE_INTERACTION=qgs3
  • low-energy interaction model: e.g., SIMTOOLS_CORSIKA_LE_INTERACTION=urqmd

  For images generated before this change, it is enough to set one single variable:

  SIMTOOLS_SIMTEL_PATH=/workdir/sim_telarray/sim_telarray

  (#1917)

• Change in naming of simtools docker images:

  • CORSIKA image is called e.g. ghcr.io/gammasim/corsika7:0.26.0-78010-avx512f
  • sim_telarray image is called e.g. ghcr.io/gammasim/sim_telarray:0.26.0-<sim_telarray version>
  • simtools-dev image is called e.g. ghcr.io/gammasim/simtools-dev:latest (safe to use the latest tag here)
  • simtools-prod image is called e.g. ghcr.io/gammasim/simtools-prod:0.26.0-corsika<corsika version>-<avx flag>-simtel<sim_telarray version>

  (#1954)

• Changed path to sim_telarray to SIMTOOLS_SIM_TELARRAY_PATH (from SIMTOOLS_SIMTEL_PATH). Requires changes by the users to their .env files. (#1957)

New Features

• Add integration tests for newly added flasher types MSFx-NectarCam, MSFx-FlashCam, LSFN-design, LSFS-design. (#1817)

• Improve handling of model parameters with output schema definition: introduce a legacy model parameter handler. (#1910)

• Allow output of flasher angular distribution tables for Lambertian distribution. (#1916)

• Refactoring the image building process (CORSIKA/sim_telarray/simtools):

  • separate images for CORSIKA and sim_telarray
  • simtools image combined both plus adds for the development image the QGSJet tables
  • main directory changed to /workdir/simulation_software/ with e.g.
    • /workdir/simulation_software/corsika
    • /workdir/simulation_software/sim_telarray
  • simulation software packages pulled from git repositories
    • CORSIKA7 from KIT gitlab and CORSIKA build configuration from CTAO gitlab
    • sim_telarray, hessio, and stdtools from CTAO gitlab
  • new build process for CORSIKA7 using coconut and configuration files

  (#1917)

• • Introduce central settings module for configuration args, db_config, and path definitions.
  • Use new settings.config``for simtel_pathusage through all modules.
  • Introduce corsika_path and corsika_he_interaction , corsika_le_interaction settings to indicate CORSIKA installation path and interaction models.
  • Ensure backwards compatibility to legacy simtools images for paths and build opts configuration files.

  (#1921)

• Allow to fill and plot simulated event distributions from both CORSIKA and sim_telarray output files.
  Add simple plotting application simtools-plot-simulated-event-distributions. (#1934)

• Use adjustText for the labels in the array layout plot to avoid overlapping labels. (#1937)

• Improved functionality to plot Cherenkov photon distributions read from CORSIKA IACT files:

  • allow to compare distributions from different files
  • additional 1D and 2D distributions with improved plotting
  • several bug fixes in relation of weighting and density calculation

  This includes a complete refactoring of the class with focus on plotting.

  (#1939)

• Add building workflow for simtools-prod image.
  Update docker files and building workflows for CORSIKA, sim_telarray, and simtools.
  Simplify package naming. (#1954)

• Add integration test run to updated production image building. Reuse existing integration tests and simplify building steps. (#1955)

• Improved version and build info handling:

  • simtools application prints at startup now the simtools, database, CORSIKA, and sim_telarray versions
  • simtools applications output full build information for log level DEBUG
  • added two new command line options:
    • --build_info to print detailed build information of the application
    • --export_build_info EXPORT_BUILD_INFO export build information to file (json or yaml format)
  • removed applications simtools-print-version (above functionality replaces it)
  • Changed path to sim_telarray to SIMTOOLS_SIM_TELARRAY_PATH (from SIMTOOLS_SIMTEL_PATH). This is consistent with the naming of the simulation software packages and the schema values.

  (#1957)

Maintenance

• Integrate the new unified flasher pulse shape parameter flasher_pulse_shape (now a 3-element list [shape, width_ns, exp_decay_ns]). (#1900)
• Allow test_generate_production_grid.py::test_convert_altaz_to_radec_and_coordinates to xfail due to intermittent network issues. (#1914)
• Add logfile pattern checks for full flasher simulations, update integration test files and add a pulse shape mapping. (#1924)
• Replace db_config by settings.config.db_config throughout all modules. (#1927)
• Moved simtel_io_event_* modules to sim_events submodule and renamed eventio_handler to file_info.
  Updated imports and class names across dependent modules to reflect generalized event file handling. (#1936)
• Add ERA (Found commented-out code) to ruff configuration (used with precommit). (#1942)
• Add PERF settings to ruff (used by pre-commit) for improved maintainability. (#1943)
• Update pylint configuration to be more aligned with SonarQ requirements. (#1944)
• Move and rename application simtools-write-array-element-positions-to-repository to simtools-maintain-simulation-model-write-array-element-positions.
  Add integration and unit tests. (#1958)
• Minor code quality improvements reported by GitHub Code QL. (#1959)
• Fixed failing unit tests for test_visualize by adding missing plt.close(). (#1967)

Simulation model

• Change default simulation models version to 0.12.0. (#1918)

v0.25.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in comparison of sim-telarray configuration files: ignore system dependent metadata values. (#1873)
• Fix thread-safe singleton check in mongo_db. Add {"maxIdleTimeMS": 10000} to close idle connections after 10s. Add debugging tool for connections. (#1899)
• Fix simulation model production table upload: full updates tables need not to search through the model history. This led to the addition of deprecated parameters back into the production. (#1913)

New Features

• Add pulse shape table writer for light emission flasher simulations. (#1851)
• Add CORSIKA executable compiled with CURVED option to docker images (corsika-curved). (#1859)
• Improve array plotting with more custom configuration options. (#1860, #1862)
• Add tests for certain strings in the CORSIKA and sim_telarray log files. (#1882)
• Verify that the request and actually simulated number of events is identical for CORSIKA and sim_telarray simulations. (#1883)
• Add tests that certain strings do not appear in the CORSIKA and sim_telarray log file (see also PR #1882) (#1885)
• Added functionality to plot mirrors and implemented auto plotting of mirrors for the reports. (#1886)
• Add test to verify that number of CORSIKA events is consistent with requested number of events. (#1890)
• Add application for pulse shape parameter derivation and move existing functionality there. (#1894)
• Add possibility to run sim_telarray only using an existing CORSIKA file with simtools-simulate-prod. (#1896)

Maintenance

• Refactor simtools-validate-file-using-schema and improve unit testing of code. (#1857)
• Introducing a new psf optimizer class and improving exception handling. (#1864)
• Update model version to ensure reproducibility if given as MAJOR.MINOR. Revert testing to exact semver version. (#1870)
• Improve description of submit_array_layouts. Fix model parameter writing of instrument for site parameters. (#1872)
• Introducing a data class and capping maximum learning rate in PSF optimization modules. (#1875)
• Remove unused command line parameter data_directory from simtools-simulate-prod. (#1891)
• Remove unused command line parameter number_of_runs from simtools-simulate-prod. Simulate always a single run number. (#1893)
• Introduce consistent naming for pedestal runs and output files: always start with pedestal... (e.g., pedestals_dark). (#1904)
• Suppress mongoDB connection close debug message in unit test. (#1906)
• Documentation is now generated for each push. Deployment changed to either the release of a version, merge to main, or manual trigger. (#1907)
• Update ignore pattern for log file inspector to ignore warnings on getpwuid obtained in AIV environment. (#1912)

Simulation model

• Merge all flasher pulse shape parameters (shape, width, exp decay) into a single parameter flasher_pulse_shape. (#1895)

v0.24.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in validation of generated sim_telarray configuration: missing assert on the outcome of the tests (tests always passed, even with differences). (#1856)
• Fixed array trigger file generation to properly combine non-hardstereo telescopes. Previously, each telescope type generated separate trigger lines, preventing software stereo triggers between different telescope types (e.g., SST+MST). Now hardstereo lines remain separate while non-hardstereo telescopes are combined into a single line, enabling cross-telescope-type triggering. (#1863)

New Features

• Add new simulation model parameter transit_time_random. (#1808)
• Implement version check for model parameters for any software type. (#1842)
• Add possibility to define new base models taking into account the correct history of models for simtools-maintain-simulation-model-add-production. (#1844)
• Add possibility to change model parameters from command line using the --overwrite_model_parameters command line parameter. (#1850)

Maintenance

• Change source of simulation model database in CI to 'main' branch of simulation model repository.
  Tests against the latest release versions for simtools release candidates (1.2.3-rc).
  Allow to configure simulation model repository branches for manually triggered workflows. (#1846)
• Review and update all XPASS unit tests. (#1855)

Simulation model

• Change default simulation models database version to 0.11.0. (#1852)

v0.23.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Bugfix in comparing model versions in integration tests to decide which tests should run.
  Bugfix in getting model parameter values with units for the case of value-unit pairs consisting of lists with None entries. (#1805)
• Fix a bug where hard stereo trigger was set every time the hard_stereo key existed in dictionary, rather than checking its value. Now it only sets hardstereo if the value is True. (#1813)
• Bugfix packing model files: model version was not taken into account correctly. (#1818)
• Bugfix in preparing sim_telarray command for shower simulations: power-law configuration (used to fill histograms) set at the wrong place. (#1825)
• Bugfix in writing sim_telarray configuration option: "-C show=all" should be last in command. (#1828)

New Features

• Add new CORSKIA / sim_telarray image build for:

  • CORSIKA 7.8010
  • Bernloehr package 1.70
  • sim_telarray 250903

  Improved downloading script and handling of QGSJet tables.

  (#1824)

• Fill telescope design types into sim_telarray metadata field for camera_config_variant and optics_config_variant. (#1826)

• Allow to highlight or grey out telescopes in layout plotting application. (#1829)

• Allow to provide integers in scientific notation in yaml configuration files. Currently only implemented for the --number_of_photons argument of calculate_incident_angles.py. (#1837)

Maintenance

• Add application control functionality for all simtools applications. (#1809)
• Adding --all_model_versions flag for calibration and software report generation. (#1815)
• Replace superlinter in linter CI. Add toml linter and shellcheck to precommit. (#1838)
• Update one function used in the tests to comply with version 2.x of pyeventio. (#1840)

v0.22.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix integration tests after introducing the usage of major.minor model versions for tests. (#1777)
• Use correct sign for flasher distance calculation with flasher z coordinate pointing towards camera. (#1784)

Documentation

• Improve documentation on DB upload. (#1775)

New Features

• Add software version range validation. Introduce software version for updated or new flasher simulation model parameters. (#1763)
• Implement simulation models production table and model parameter update using the simtools-maintain-simulation-model-add-production application. (#1765)
• Schema for info.yml files used in simulation models. (#1790)
• Add flat-fielding devices for all telescope types. (#1798)
• Add new application simtools-db-upload-model-repository for simplified upload of repository data to the database. (#1801)
• Add flasher exponential decay pulse shape parameter. (#1802)

Maintenance

• Linked parameter plots to parameter comparison reports. (#1661)
• Added CLI argument to flexibly change PSF containment percentage as needed. (#1766)
• Remove 'LATEST' options when setting the DB_SIMULATION_MODEL_VERSION configuration option (require explicit setting of the DB version). (#1785)
• Introduce a simpler naming for the database (CTAO-Simulation-Model instead of CTAO-Simulation-ModelParameters). (#1787)
• Remove use_plain_output option and generalize output direction generation. (#1792)
• Simplify generation of production image and remove outdated build-corsika-simtelarray target. (#1794)
• Increase robustness for unit and integration tests when cloning remote Git repositories. Add a 3x retry. (#1796)
• Split CORSIKA/sim_telarray and simtools images into two. (#1797)
• Change minimum python version requirement to 3.12 (following CTAO recommendation). (#1799)
• Remove 'instrument:type' and 'instrument:site' in model parameter schema files. (#1803)

v0.21.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix output file error when running application simtools-print-version. (#1774)
• Bugfix in configuring hard-stereo array triggers (hard_stereo should be hardstereo in array_trigger.dat). (#1780)

New Features

• Implemented gradient descent minimisation for psf parameter optimisation, KS statistic as a sanity check, and monte carlo error on the simulated psf. (#1717)
• Add functionality to work with delta-type production tables of the simulation model for patch updates. (#1758)
• Automatic retrieval of the latest patch version for the simulation model.
  Allow --model_version=6.0 being resolved to the newest version. (#1768)

v0.20.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix warning messages in simtel metadata reader to avoid the log-file testing routines to fail. (#1712)
• Developer image is using an old base image with CORSIKA and sim_telarray. Fix naming of containers used in developer image to ensure the most recent are used. (#1716)
• Introduce DB_SIMULATION_MODEL_VERSION to fix instability of environment when introducing new model parameters. (#1729)
• Fix reduce complexity issue raised in sonar in plot simtel events. (#1735)
• Fix bug in schema file for model parameters regarding schema version. Improved printing on error. (#1742)

New Features

• Add new application to calculate telescope and array trigger rates from reduced event lists. (#1685)
• Add application to calculate photon incident angles at the camera plane. (#1687)
• Add sim_telarry executable compiled with debug options to containers. Allows to generate additional output columns for e.g. sim_telarray ray tracing. (#1715)
• Add / update nsb_spectrum model parameter and calculate integrated NSB rate. (#1718)
• Add avx512f and sse4 vector optimization images for production images. (#1741)

Maintenance

• Remove calculate trigger rate application and associated code. Replaced by simtools-derive-trigger-rate (see PR #1685). (#1693)
• Improve robustness of image building and downloading autoconf. (#1722)
• Improvement of CI-workflow efficiency and documentation. (#1723)
• Improve handling of secrets in CI workflows. (#1728)
• Add new field production_table_name to production table schema. (#1756)
• Remove Codecov from CI (covered by SonarQube). (#1759)

v0.19.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix bug in reading list of array elements from file. Only layout names were used and the list was read not from file but from the model parameters DB. (#1658)
• Ensure that run numbers start at 1 for cases no offset if given (HT Condor job submission). (#1686)
• Fix array trigger writing for single telescope simulations. (#1690)
• Bugfix in shower core calculation in shower coordinates for reduced event data. (#1703)

New Features

• Add additional plots for the derivation of CORSIKA limits. (#1632)
• Add a warning to derived corsika limit if they are close to the limits of the simulations. (#1643)
• Add an application that merges derived corsika simulation limits for each available grid point. (#1649)
• Add curvature radius to testeff command. User for calculation of NSB rates per pixel. (#1651)
• Allow to run applications in container environments using simtools-run-application. Introduce separate module (simtools-runner) and improve support for workflow running. (#1657)
• Add application to update, modify, and compare production tables.
  Improve documentation for simulation repository maintenance. (#1660)
• Add application to simulate calibration events (pedestals, dark pedestals, flasher). (#1663)
• Add functionality to simtools-validate-camera-efficiency to write out model parameter JSON with nsb_pixel_rate. (#1665)
• Add functionality to compare derived model parameters with DB values to integration tests (examples for simtools-validate-camera-efficiency). (#1666)
• Add log file inspector for integration tests to fail tests when errors or runtime warnings are reported. (#1674)
• Add application to simulate flasher with the light emission package. (#1676)
• Add sim_telarray model files to output of each production run. This asserts reproducibility and simplifies debugging. (#1689)
• Add MongoDB compound indices to improve query efficiency. (#1691)
• Add possibility to set random seed for random focal length settings in simtools-derive-mirror-rnda. (#1702)

Maintenance

• Apply more consistent key-name convention with lower-case snake_make to application workflow configurations. (#1636)
• Clarify input data to derivation of CORSIKA limits with simtools-production-derive-corsika-limit. (#1641)
• Change metadata to lower-case snake make. More consistent usage of schema versions. (#1645)
• Add missing plotting configurations for tabular data. (#1647)
• Plots are now auto-generated alongside reports. (#1653)
• Refactoring of ASCII input io operations. (#1667)
• Add a generic writer for structured data for JSON and YAML format. (#1668)
• Simplification of legend_handlers class. (#1670)
• Change units of shower- and pointing directions in reduced event data from radians to degrees. (#1672)
• Generalization of reduced event-data histogram filling (moved from CORSIKA limits derivation modules to generic module). (#1675)
• Fix hadolint issue and using parameter expansion for replacement in Dockerfile. (#1680)
• Refactor PSF parameter optimization workflow and improve output. Added a flag to export best parameters as JSON files. (#1681)
• Revision of main repository readme file and removed duplications. (#1684)
• Remove DB access of unit test for the docs_read_parameters module. (#1688)
• Maintenance pass through integration test to remove duplicated tests and improve test efficiency. (#1694)
• Add new sonar configuration (URL) (#1695)
• Improve code quality by addressing SonarQube reliability issues. (#1697)
• Unit tests will now fail in the CI for warnings. (#1699)
• Fix some security hotspots reported by SonarQube. (#1700)
• Improve robustness of Linter CI by caching pre-commit installation. (#1704)

Simulation model

• Addition of FADC long-sum model parameters. (#1659)

v0.18.0

This release of simtools, the simulation toolkit for the CTA Observatory, includes the following main changes:

Bugfixes

• Run documentation generation CI on release to obtain current version of docs. (#1607)
• Fix authentication errors in script to setup a local mongoDB. (#1610)

Documentation

• Reorganisation and improvement of the documentation in the GitHub pages. (#1611)

New Features

• Use CTAO telescope names when writing out telescope list in reduced event data tables. (#1616)
• Allow CORSIKA limits to use layouts defined in simulation models database. (#1619)
• Add application / API to plot tabulated model parameters using default plotting configurations defined in model parameters schemas. (#1628)
• Allow to use current prod6 simulations for reduced event data generation. (#1631)
• Add common array elements ID. Add list with triggered common telescope IDs to reduced event data tables. (#1637)

Maintenance

• Removed unused ‘critical’ log level. (#1604)
• Add JSON checker to pre-commit linter. (#1615)
• Update test files with proton simulations (simtel output) and add provenance information. (#1617)
• Improve linter CI to print file containing non-ascii characters. (#1620)
• Replace file lists tests/resources/simtel_output_files.txt used for testing by explicitly listing of files. (#1629)
• Add default plotting configuration for tabular data. Introduce consistent lower-cases for keys in plotting configuration schema. (#1635)

v0.17.0

Release 0.17.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix data type of 'dsum_threshold' parameter. (#1557)
• Add missing neutrons to eventio particle IDs. (#1584)
• Fix suffix for sim_telarray output files: no correctly simtel.zst (was .zst only) (#1588)

Documentation

• Change documentation theme to pydata and add version switcher. (#1560)
• Update documentation on how to run jobs using HT Condor. (#1563)

New Features

• Add pixel layout plotting. (#1541)
• Enable docker-image generation for vector-optimized CORSIKA 7.8. (#1550)
• Add the sequential option to simulations using multipipe. With this option the CORSIKA and sim_telarray instances run in sequential order as far as possible. This mode is useful particularly on the grid, where typically we request a single core per job. (#1561)
• Add reading and conversion of primary_id from sim_telarray output. (#1570)
• Update simtools-generate-simtel-event-data to use astropy tables. (#1573)
• Generalize table reading and writing. Add application to merge tables. (#1580)
• Write reduced event data file as part of the cleanup stage of the simtools-simulate-prod application. (#1588)
• Use patches for optimized CORSIKA from gitlab repository instead of cloud download. (#1589)
• New utility simtools-verify-simulation-model-production-tables to verify completeness of model parameter repository. (#1592)
• Add custom copilot instructions to support AI used by developers. (#1594)
• Add new application simtools-submit-array-layouts to submit new array-layouts definition and validate the telescopes listed. (#1596)
• Improve layout plotting; adding plotting of background layout. Add plotting of all layouts. (#1597)
• Improve printout for assertion error for testing. (#1598)
• Add production version and build options to sim_telarrary metadata. (#1601)

Maintenance

• Add unit tests to increase coverage in interpolation handler. (#1504)
• Docker file improvements: linter application; bugfix in extra flags for optimized CORSIKA compilation. (#1544)
• Include stdout and stderr in error message of integration test. (#1555)
• Add pytest-retry package for 'flaky test'. (#1558)
• Reduce number of log-messages (INFO) for uploaded model parameters to DB. (#1559)
• Replace the "run_number_start" argument with "run_number_offset" (clearer) and re-introduce the "run_number" argument to allow specifying a run number (to which the offset is added). (#1562)
• Use telescope position files for setup in illuminator application and set correct obs level. (#1566)
• Improve the names of the various simulation output files. They now follow the current convention in current productions, they contain a bit more information (e.g., the model version where it wasn't there before), and they are more consistent with each other. (#1568)
• Improved interface and interpolation routines to derive production configuration. This creates a pipeline for lookup table generation, production grid generation, creating a grid with limits, which is then further complemented by the production statistics per grid point. (#1569)
• Fix unit test to run with astropy 7.1 (and earlier versions) (see #1574). (#1575)
• Simplified Sonar setup by moving properties into GitHub workflow. (#1578)
• Introduce simplified Codecov configuration. (#1579)
• Copy the CORSIKA log in binary mode instead of text mode. (#1582)
• Protect against non-unicode characters in CORSIKA log file. (#1583)
• Remove obsolete code on submission systems in JobManager. (#1586)
• Change dummy trigger thresholds to easier-to-spot values. (#1591)
• Fix duplication of some sim_telarray test files in './tests/resources' and corresponding test fixtures. (#1593)