Merge pull request #753 from migueldiascosta/eb450

release EasyBuild v4.5.0

.readthedocs.yml

@@ -18,3 +18,4 @@ python:
   install:
       - method: pip
         path: .
+      - requirements: docs/requirements.txt

docs/Changelog.rst

@@ -5,6 +5,7 @@ Changelog for EasyBuild documentation
 
 (for EasyBuild release notes, see :ref:`release_notes`)
 
+* **release 20211029.01** (`Oct 29th 2021`): update release notes for EasyBuild v4.5.0 (see :ref:`release_notes_eb450`)
 * **release 20210907.01** (`Sep  7th 2021`): update release notes for EasyBuild v4.4.2 (see :ref:`release_notes_eb442`)
 * **release 20210706.01** (`Jul  6th 2021`): update release notes for EasyBuild v4.4.1 (see :ref:`release_notes_eb441`)
 * **release 20210602.01** (`Jun  2nd 2021`): update release notes for EasyBuild v4.4.0 (see :ref:`release_notes_eb440`)

docs/Common-toolchains.rst

@@ -182,6 +182,8 @@ Component versions in ``foss`` toolchain
 +----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+
 | ``2021a``      | May '21     | 2.36.1     | 10.3.0    | 4.1.1      | 3.0.4       | 0.3.15     | (incl. with OpenBLAS) | 2.1.0        | 3.3.9       |
 +----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+
+| ``2021b``      | Oct '21     | 2.37       | 11.2.0    | 4.1.1      | 3.0.4       | 0.3.18     | (incl. with OpenBLAS) | 2.1.0        | 3.3.10      |
++----------------+-------------+------------+-----------+------------+-------------+------------+-----------------------+--------------+-------------+
 
 *(components marked with* * *were patched)*
 
@@ -222,6 +224,8 @@ Component versions in ``intel`` toolchain
 +----------------+-------------+-------------+-----------+--------------------+-------------+--------------+
 | ``2021a``      | May'21      | 2.36.1      | 10.3.0    | 2021.2.0           | 2021.2.0    | 2021.2.0     |
 +----------------+-------------+-------------+-----------+--------------------+-------------+--------------+
+| ``2021b``      | Oct'21      | 2.37        | 11.2.0    | 2021.4.0           | 2021.4.0    | 2021.4.0     |
++----------------+-------------+-------------+-----------+--------------------+-------------+--------------+
 
 ``(*)``: This toolchain is deprecated, see :ref:`deprecated_easyconfigs_toolchains` for more information.
 

docs/Deprecated-easyconfigs.rst

@@ -64,35 +64,61 @@ Deprecated toolchains
 Overview of deprecated toolchains:
 
 * :ref:`deprecated_easyconfigs_toolchains_foss`
+* :ref:`deprecated_easyconfigs_toolchains_gcc`
+* :ref:`deprecated_easyconfigs_toolchains_gcccuda`
 * :ref:`deprecated_easyconfigs_toolchains_gompi`
 * :ref:`deprecated_easyconfigs_toolchains_goolf`
 * :ref:`deprecated_easyconfigs_toolchains_ictce`
 * :ref:`deprecated_easyconfigs_toolchains_intel`
+* :ref:`deprecated_easyconfigs_toolchains_iomkl`
 
 
 .. _deprecated_easyconfigs_toolchains_foss:
 
-``foss`` toolchain
-~~~~~~~~~~~~~~~~~~
+``foss`` and ``fosscuda`` toolchain
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
+
+The oldest versions of the ``foss`` and ``fosscuda`` toolchains have been deprecated,
+which currently includes any version older than ``foss/2019a`` and ``fosscuda/2019a``.
+
+
+.. _deprecated_easyconfigs_toolchains_gcc:
+
+``GCCcore`` and ``GCC`` toolchains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
+
+The oldest versions of the ``GCCcore`` and ``GCC`` toolchains have been deprecated,
+which currently includes any version older than ``8.0``.
+
+
+.. _deprecated_easyconfigs_toolchains_gcccuda:
 
-* *deprecated since:* EasyBuild v3.9.4
-* *will be archived in:* EasyBuild v4.0.0
+``gcccuda`` toolchain
+~~~~~~~~~~~~~~~~~~~~~
 
-The oldest versions of the ``foss`` toolchain have been deprecated,
-which currently includes any version older than ``foss/2016a``.
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
+
+The oldest versions of the ``gcccuda`` toolchains have been deprecated,
+which currently includes any version older than ``gcccuda/2019a``.
 
 
 .. _deprecated_easyconfigs_toolchains_gompi:
 
-``gompi`` toolchain
-~~~~~~~~~~~~~~~~~~~
+``gompi`` and ``gompic`` toolchains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-* *deprecated since:* EasyBuild v3.8.0
-* *will be archived in:* EasyBuild v4.0.0
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
 
-Versions of the ``gompi`` toolchain that were used as a subtoolchain for a deprecated toolchain
-have also been deprecated; this includes ``gompi`` toolchain versions that match ``1.*``, and any version
-older than ``gompi/2016a`` (the latter only since EasyBuild v3.9.4).
+Versions of the ``gompi`` and ``gompic`` toolchains that were used as a subtoolchain for a deprecated toolchain
+have also been deprecated; this includes versions older than ``gompi/2019a`` and ``gompic/2019a``.
 
 
 .. _deprecated_easyconfigs_toolchains_goolf:
@@ -101,7 +127,7 @@ older than ``gompi/2016a`` (the latter only since EasyBuild v3.9.4).
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 * *deprecated since:* EasyBuild v3.8.0
-* *will be archived in:* EasyBuild v4.0.0
+* *archived in:* EasyBuild v4.0.0
 
 The ``goolf`` and ``goolfc`` toolchains have been deprecated,
 since they are superseded by the :ref:`common_toolchains_foss` and ``fosscuda`` toolchains, respectively.
@@ -116,7 +142,7 @@ except that ``binutils`` is also included as a companion to ``GCC(core)`` in the
 ~~~~~~~~~~~~~~~~~~~
 
 * *deprecated since:* EasyBuild v3.8.0
-* *will be archived in:* EasyBuild v4.0.0
+* *archived in:* EasyBuild v4.0.0
 
 The ``ictce`` toolchain has been deprecated, since it is superseded by the :ref:`common_toolchains_intel`.
 
@@ -127,17 +153,28 @@ for the Intel compilers) in the ``intel`` toolchain.
 
 .. _deprecated_easyconfigs_toolchains_intel:
 
-Oldest versions of the ``iccifort``, ``iimpi`` and ``intel`` toolchains
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``iccifort``, ``iimpi``, ``iimpic``, ``intel``, and ``intelcuda`` toolchains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-* *deprecated since:* EasyBuild v3.8.0
-* *will be archived in:* EasyBuild v4.0.0
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
+
+The oldest versions of the ``iccifort``, ``iimpi`` and :ref:`common_toolchains_intel` have been deprecated.
+
+Deprecated versions include:
+
+* ``iccifort`` versions older than ``2019.0``
+* ``iimpi`` and ``iimpic`` versions older than ``2019a``
+* ``intel`` and ``intelcuda`` versions older than ``2019a``
+
+
+.. _deprecated_easyconfigs_toolchains_iomkl:
 
-The oldest versions of the ``iccifort``, ``iimpi`` and :ref:`common_toolchains_intel` have been deprecated,
-since they are no longer considered relevant for recent systems.
+``iompi``, ``iompic``, ``iomkl``, and ``iomklc`` toolchains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-More specifically, deprecated versions include:
+* *deprecated since:* EasyBuild v4.5.0
+* *will be archived in:* EasyBuild v5.0.0
 
-* ``iccifort`` versions older than ``2016.1.150``
-* ``iimpi`` versions older than ``2016.01``, except version ``8.1.5-*``
-* ``intel`` versions older than ``2016a``
+The oldest versions of the ``iompi``, ``iompic``, ``iomkl`` and ``iomklc`` toolchains have been deprecated,
+which currently includes any version older than ``2019a``.

docs/Experimental_features.rst

@@ -14,3 +14,4 @@ Currently enabled experimental features include:
  * support for easyconfig files in YAML syntax (see :ref:`easyconfig_yeb_format`)
  * support for generating container recipes & images (see :ref:`containers`)
  * support for using easystack files (see :ref:`easystack`)
+ * support for installing extensions in parallel (see :ref:`installing_extensions_in_parallel`)

docs/Installation.rst

@@ -503,6 +503,8 @@ Optional Python packages
   SVN repository for easyconfig files;
 * `python-graph-dot <https://pypi.python.org/pypi/python-graph-dot/>`_,
   only needed for building nice-looking dependency graphs using ``--dep-graph *.dot``
+* `Rich <https://pypi.org/project/rich/>`_,
+  only needed to let ``eb`` produce rich output, like :ref:`progress_bars`;
 
 Sources
 -------

docs/Installing_extensions_in_parallel.rst

@@ -0,0 +1,109 @@
+.. _installing_extensions_in_parallel:
+
+Installing extensions in parallel (experimental!)
+=================================================
+
+Since EasyBuild v4.5.0 (initial) support is available for installing extensions in parallel,
+that is to run the installation command for multiple extensions that are ready to install at the same time
+in the background, to exploit the availability of multiple cores.
+
+**Note: this is an** :ref:`experimental feature <experimental_features>`.
+**Some of the mentioned functionality may be subject to change or be prone to errors.**
+
+.. contents::
+    :depth: 3
+    :backlinks: none
+
+.. _installing_extensions_in_parallel_configuration:
+
+Configuration
+-------------
+
+To let EasyBuild install extensions in parallel, the ``parallel-extensions-install`` configuration option must be
+enabled.
+
+In addition, since the support for installing extensions in parallel is an experimental feature, the ``experimental``
+configuration option must be also be enabled (see also :ref:`experimental_features`).
+
+How many extension installations should be started (at maximum) at the same time is controlled by
+the ``parallel`` configuration option.
+
+
+.. _installing_extensions_in_parallel_requirements:
+
+Requirements
+------------
+
+For EasyBuild to be able to install a list of extensions in parallel, two requirements must be met:
+
+* The easyblock(s) that are used for installing the extensions must have support for determining
+  which installed extensions are required to fulfill the dependencies for a particular extension.
+  This is done via the ``required_deps`` property method.
+* EasyBuild must be able to start a single shell command (asynchronously, so it runs in the background)
+  to perform the installation of the extension via the ``run_async`` method,
+  and check whether the command has completed via the ``async_cmd_check`` method.
+
+For an example of how ``required_deps``, ``run_async``, and ``async_cmd_check`` can be implemented,
+see the `RPackage
+<https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/rpackage.py>`_ easyblock.
+
+.. _installing_extensions_in_parallel_caveats_known_issues:
+
+Caveats and Known issues
+------------------------
+
+*(last update: EasyBuild v4.5.0)*
+
+There are a couple of caveats and known issues with installing extensions in parallel,
+which we hope to resolve in future EasyBuild releases.
+
+This is also why the support for installing extensions in parallel is currently marked as an experimental feature.
+
+.. _installing_extensions_in_parallel_caveats_known_issues_only_r:
+
+Only works for R extensions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The support for installing extensions in parallel currently only works for R extensions,
+since only the ``RPackage`` easyblock (which is used to install R packages as extensions) implements
+the necessary methods (see :ref:`installing_extensions_in_parallel_requirements`).
+
+Supporting other types of extensions (Python, Perl, etc.) requires that the corresponding easyblocks that
+are used to install those extensions (like ``PythonPackage``, ``PerlModule``) are enhanced to support determining
+required dependencies and starting the installation command asynchronously.
+
+.. _installing_extensions_in_parallel_caveats_known_issues_exts_list:
+
+List of extensions must be self-contained
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The EasyBuild framework currently enforces that all required dependencies for a particular extension
+are included in the list of extensions that is being installed (specified via the ``exts_list`` easyconfig
+parameter). Extensions provided by dependencies are not taken into account (yet).
+
+As a result, trying to install an
+`R-bundle-Bioconductor <https://github.com/easybuilders/easybuild-easyconfigs/tree/main/easybuild/easyconfigs/r/R-bundle-Bioconductor>`_
+easyconfig with ``parallel-extensions-install`` enabled
+will result in an error reporting missing required dependencies.
+
+
+.. _installing_extensions_in_parallel_caveats_known_issues_skip:
+
+Skipping of installed extensions is still done sequentially
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Checking which extensions are already installed for which the installation can be skipped
+when ``--skip`` (see :ref:`partial_installation_skip`) is being used is still done sequentially.
+
+Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.
+
+
+.. _installing_extensions_in_parallel_caveats_known_issues_sanity_check:
+
+Sanity check for extensions is still run sequentially
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The sanity checks that are done for each extension are still done through a sequential loop over the
+list of extensions.
+
+Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.