Merge pull request #615 from migueldiascosta/master

release EasyBuild v4.2.0

docs/Changelog.rst

@@ -5,6 +5,11 @@ Changelog for EasyBuild documentation
 
 (for EasyBuild release notes, see :ref:`release_notes`)
 
+* **release 20200414.01** (`Apr 14th 2020`):
+
+  * document new EasyBuild locking mechanism (see :ref:`locks`)
+  * document support for creating index files (see :ref:`easyconfigs_index`)
+  * update release notes for EasyBuild v4.2.0 (see :ref:`release_notes_eb420`)
 * **release 20200316.01** (`Mar 16th 2020`): update release notes for EasyBuild v4.1.2 (see :ref:`release_notes_eb412`)
 * **release 20200116.01** (`Jan 16th 2020`): update release notes for EasyBuild v4.1.1 (see :ref:`release_notes_eb411`)
 * **release 20191204.01** (`Dec 4th 2019`): update release notes for EasyBuild v4.1.0 (see :ref:`release_notes_eb410`)

docs/Easyconfigs_index.rst

@@ -0,0 +1,175 @@
+.. _easyconfigs_index:
+
+Using an index to speed up searching for easyconfigs
+====================================================
+
+EasyBuild often needs to search for :ref:`easyconfig_files` (or accompanying files like patches),
+either based on command line arguments (like the name of an easyconfig file)
+or options to the ``eb`` command (like ``--search``, see :ref:`searching_for_easyconfigs`),
+or to resolve dependencies for a particular easyconfig file that was parsed already.
+
+Searching for easyconfig files or patches may take a while, since it can potentially involve weeding through
+thousands of files, which may be located on a shared filesystem (where metadata-intensive operations like
+file searching can be rather slow).
+
+If EasyBuild turns out to be sluggish when searching for easyconfig files in your setup,
+using an *index* file could make a big difference.
+
+.. note:: Support for index files was implemented in EasyBuild version 4.2.0 .
+
+
+.. contents::
+    :depth: 3
+    :backlinks: none
+
+
+.. _easyconfigs_index_create:
+
+Creating an index (``--create-index``)
+--------------------------------------
+
+``eb --create-index`` can be used to create an index file for a particular directory
+that holds a (large) set of easyconfig files.
+
+The index file (a hidden file named ``.eb-path-index``) will be created in the specified directory.
+It will contain a list of (relative) paths for all files in that directory,
+along with some metadata: the time at which the index file was created
+and a timestamp that indicates how long the index is considered to be valid (see :ref:`easyconfigs_index_max_age`).
+
+.. note::
+  Make sure to create an index for the correct path.
+
+  The search for easyconfig files performed by EasyBuild will *not* recurse into subdirectories of the locations
+  it considers (see :ref:`robot_search_path`), other than those with a name matching the
+  software for which it is trying to find an easyconfig file (like ``t/TensorFlow/`` when searching for an
+  easyconfig file for ``TensorFlow``).
+
+  Hence, if the directory housing your easyconfig files has an ``easybuild/easyconfigs`` subdirectory
+  (for example, a working copy of the ``easybuild-easyconfigs`` repository), create the index *in* that
+  subdirectory, rather than in the higher-level directory.
+
+
+.. _easyconfigs_index_create_example:
+
+Example of creating an index
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code::
+
+  $ eb --create-index $HOME/easybuild-easyconfigs/easybuild/easyconfigs
+  == temporary log file in case of crash /tmp/eb-tUu6_w/easybuild-SKBnVO.log
+  Creating index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs...
+  == found valid index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
+  Index created at /home/example/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index (738 files)
+
+  $ head -n 5 $HOME/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index
+  # created at: 2020-04-13 14:19:57.008981
+  # valid until: 2020-04-20 14:19:57.008981
+  a/Arrow/Arrow-0.16.0-intel-2019b-Python-3.7.4.eb
+  b/Boost/Boost-1.71.0-gompi-2019b.eb
+  b/bokeh/bokeh-1.4.0-foss-2019b-Python-3.7.4.eb
+
+
+.. note::
+  The "``found valid index ...``" message being printed occurs because EasyBuild tries to load the index file
+  right after creating it, as a sanity check.
+
+
+.. _easyconfigs_index_update:
+
+Updating an existing index (``--create-index --force``)
+-------------------------------------------------------
+
+To update an existing index, you can use ``--create-index --force``.
+
+Without using ``--force``, EasyBuild will refuse to overwrite the existing index file::
+
+  $ eb --create-index $HOME/easybuild-easyconfigs/easybuild/easyconfigs
+  == temporary log file in case of crash /tmp/eb-tUu6_w/easybuild-SKBnVO.log
+  Creating index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs...
+  ERROR: File exists, not overwriting it without --force: /home/example/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index
+
+
+.. _easyconfigs_index_use:
+
+Using index files
+-----------------
+
+EasyBuild will automatically pick up and use any index file that it runs into while searching for
+easyconfig files or patches. If an index file is found, it will be preferred over walking through
+the directory tree to check for the target file, which is likely to significantly speed up the search operation.
+
+When a (valid) index file is found for a particular path, a message will be printed mentioning "``found valid index
+for...``"::
+
+  $ eb --search TensorFlow-2.1.0-foss-2019b
+  == found valid index for /home/example/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
+ * /home/example/easybuild-easyconfigs/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb
+
+
+.. _easyconfigs_index_ignore:
+
+Ignoring indices (``--ignore-index``)
+-------------------------------------
+
+One potential issue with having an index in place is that it may get outdated:
+new files may have been added to the directory since the index was created or last updated.
+
+If updating the indexes is not an option (see :ref:`easyconfigs_index_update`),
+you can instruct EasyBuild to ignore any existing indices using the ``--ignore-index``
+configuration option.
+
+The only downside of this option is that searching for easyconfig files may be significantly slower.
+Any existing index files are left untouched (they will *not* be updated or removed).
+
+
+.. _easyconfigs_index_max_age:
+
+Controlling how long the index is valid (``--index-max-age``)
+-------------------------------------------------------------
+
+When creating an index file, you can specify how long the index should be considered valid.
+
+Using the ``--index-max-age`` configuration option, you can specify how long after the creation time
+the index remains valid (in seconds).
+
+By default, EasyBuild will consider index files to remain valid for 1 week (7 * 24 * 60 * 60  = 604,800 seconds).
+
+To create an index that *always* remains valid (never expires), use zero (``0``) as value for ``--index-max-age``::
+
+  $ eb --create-index --index-max-age=0 $HOME/easybuild-easyconfigs/easybuild/easyconfigs
+
+  $ head -n 2 $HOME/easybuild-easyconfigs/easybuild/easyconfigs/.eb-path-index
+  # created at: 2020-04-13 15:10:07.173191
+  # valid until: 9999-12-31 23:59:59.999999
+
+.. note:: Trust us here, December 31st 9999 is the end of times. Better get prepared.
+
+
+.. _easyconfigs_index_release:
+
+Index included with EasyBuild releases
+--------------------------------------
+
+Each EasyBuild release (since EasyBuild v4.2.0) comes with an index file for the easyconfig (and patch) files
+that are included with that release.
+
+Hence, you only need to use ``--create-index`` to create/update the index file for any additional directories
+with easyconfig files you may have on the side (and only if searching those easyconfig files is rather slow).
+
+
+.. _easyconfigs_index_should_use:
+
+Should I create an index?
+-------------------------
+
+Whether or not you should create an index file for your directories housing additional easyconfig files depends on a number of factors, including:
+
+* how often files are added and/or removed in those directories, since files listed in the index are assumed to be there and any files not included in the index will be overlooked by EasyBuild when it's searching for files;
+
+* the filesystem on which those directories are located, since an index file will only make a significant difference on filesystems where metadata-intensive operations are relatively slow;
+
+* how many files there are in those directories, since performance benefits will only be apparent if the number if files is sufficiently large;
+
+.. note:: Keep in mind that creating an index implies also updating it frequently,
+          to ensure that EasyBuild will take all available files in account.

docs/Including_additional_Python_modules.rst

@@ -85,6 +85,9 @@ Generic easyblocks are expected to be located in a directory named ``generic``.
 To verify that the easyblocks you included are indeed being picked up, ``--list-easyblocks=detailed`` can be used
 (see also :ref:`list_easyblocks`).
 
+Since EasyBuild 4.2.0, easyblocks from a pull request on GitHub can also be included,
+using ``--include-easyblocks-from-pr`` (see :ref:`github_include_easyblocks_from_pr`).
+
 Example
 ~~~~~~~
 

docs/Integration_with_GitHub.rst

@@ -353,6 +353,38 @@ in that same pull request for netCDF, WRF, ...::
 Again, note that locally available easyconfigs that are required to resolve dependencies are being picked up as needed.
 
 
+.. _github_include_easyblocks_from_pr:
+
+Using easyblocks from pull requests (``--include-easyblocks-from-pr``)
+----------------------------------------------------
+
+*(supported since EasyBuild v4.2.0)*
+
+Via the ``--include-easyblocks-from-pr`` command line option, easyblocks that are added or
+modified by a particular pull request to the `easybuild-easyblocks GitHub repository
+<https://github.com/easybuilders/easybuild-easyblocks>`_ can be used (regardless of whether the pull request is merged
+or not).
+
+This can be useful to employ easyblocks that are not available yet in the active EasyBuild installation,
+or to test new contributions by combining ``--include-easyblocks-from-pr`` with ``--from-pr`` and ``--upload-test-report``
+(see :ref:`github_upload_test_report`).
+
+When ``--include-easyblocks-from-pr`` is used, EasyBuild will download all modified easyblocks to a temporary
+directory before processing them. Just like with ``--include-easyblocks`` (see :ref:`include_easyblocks`),
+the easyblocks that are included are preferred over the ones included in the EasyBuild installation.
+
+For example, to use the LAMMPS easyblock contributed via `easyblocks pull request #1964 
+<https://github.com/easybuilders/easybuild-easyblocks/pull/1964>`_ together with the LAMMPS v7Aug2019 easyconfigs contributed via 
+`easyconfigs pull request #9884 <https://github.com/easybuilders/easybuild-easyconfigs/pull/9884>`_::
+
+    $ eb --from-pr 9884 --include-easyblocks--from-pr 1964 --list-easyblocks=detailed
+    == temporary log file in case of crash /tmp/eb-Eq2zsJ/easybuild-1AaWf8.log
+    EasyBlock (easybuild.framework.easyblock)
+    ...
+    |   |   |-- EB_LAMMPS (easybuild.easyblocks.lammps @ /tmp/included-easyblocks-rD2HEQ/easybuild/easyblocks/lammps.py)
+    ...
+
+
 .. _github_upload_test_report:
 
 Uploading test reports (``--upload-test-report``)

docs/Locks.rst

@@ -0,0 +1,102 @@
+.. _locks:
+
+Locks to prevent duplicate installations running at the same time
+=================================================================
+
+Easybuild creates a lock before starting the installation of an easyconfig file,
+to avoid a collision between multiple installations running at the same time.
+
+If an EasyBuild session tries to install an easyconfig file when a lock for that installation already exists,
+it will be automatically aborted with an error like "``Lock ... already exists, aborting!``".
+
+.. note:: Locking of installations was implemented in EasyBuild version 4.2.0 .
+
+
+.. contents::
+    :depth: 3
+    :backlinks: none
+
+.. _locks_implementation:
+
+Locking implementation details
+------------------------------
+
+Easybuild will create a lock when starting an installation if no corresponding lock exists yet,
+regardless of whether the installation was requested explicitly or is performed to resolve a dependency.
+
+The name of the lock corresponds to the *full* path of the installation directory, with slashes (``/``) and
+dashes (``-``) replaced by underscores (``_``), and with an additional ``.lock`` added at the end.
+
+Locks are created in the :ref:`locks_dir`.
+
+The lock created by EasyBuild is an empty directory (rather than a file),
+because that can be created more atomically on modern filesystems.
+
+For example, if ``OpenFOAM-7-foss-2019b.eb`` is being installed to ``/apps/easybuild/software``,
+an empty directory that serves as a lock for this installation will be created at
+``/apps/easybuild/software/.locks/_apps_easybuild_software_OpenFOAM_7_foss_2019b.lock``
+(assuming the default :ref:`locks_dir` is used).
+
+A lock is automatically removed by Easybuild when the installation ends, regardless of whether the installation
+was successful or not. Therefore, new installations of the same easyconfig will be aborted in case of:
+
+* another installation for the same easyconfig is in progress;
+* a previous installation of the same easyconfig was abruptly interrupted;
+
+
+.. _locks_removing:
+
+Removing locks
+--------------
+
+If a previous installation was abruptly interrupted and a lock was left in place,
+it can be easily removed using the ``rmdir`` command (since the lock is actually an empty directory).
+
+
+.. _locks_configuration_options:
+
+Configuration options related to installation locks
+---------------------------------------------------
+
+The behaviour of the locking mechanism in Easybuild can be tuned with the following configuration options:
+
+.. _locks_ignore:
+
+Ignoring locks (``--ignore-locks``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Using the ``--ignore-locks`` configuration option, you can instruct EasyBuild to ignore any existing locks.
+Locks that exist are left untouched, even if the installation completes successfully.
+
+**Use this with caution, since installations may be (partially) overwritten if another EasyBuild session is also
+performing those installations!**
+
+
+.. _locks_wait:
+
+Waiting for locks to be removed (``--wait-on-lock``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Using the ``--wait-on-lock`` configuration option, you can change how EasyBuild deals with existing locks,
+by specifying how frequently EasyBuild should check whether an existing lock was removed. By specifying a non-zero value ``S``,
+you can indicate how many seconds EasyBuild should wait before checking again whether the lock is still in place.
+
+.. note:: EasyBuild will wait indefinitely for an existing lock to be removed if ``--wait-on-lock`` is set to a non-zero value...
+
+  If the lock is never removed, the EasyBuild session will never terminate; it will keep checking every ``S`` seconds whether the lock is still in place.
+
+By default, EasyBuild will *abort* the installation with an error like "``Lock ... already exists, aborting!``"
+if a corresponding lock already exists, which is equivalent to setting ``--wait-on-lock`` to zero (``0``),
+implying that no waiting should be done at all.
+
+
+.. _locks_dir:
+
+Locks directory
+---------------
+
+If desired, an alternate location where locks should be created and checked for can be specified via the ``--locks-dir`` configuration option.
+
+.. note:: Keep in mind that a path on a *shared* filesystem should be used, to ensure that active EasyBuild sessions running on different systems use the same locks directory.
+
+By default, locks are created in a hidden subdirectory ``.locks`` in the top-level ``software`` installation directory; that is, the ``software`` subdirectory of the ``installpath`` configuration setting of the active EasyBuild session.