Merge branch 'master' of github.com:hpcugent/easybuild

docs/Changelog.rst

@@ -3,6 +3,45 @@
 Changelog for EasyBuild documentation
 -------------------------------------
 
+* **release 20150430.01** (`Apr 30th 2015`): updated documentation for EasyBuild v2.1.0
+
+  * also cover extensions in page on concepts and terminology (see :ref:`extensions`)
+  * add documentation on :ref:`partial_installations`, covering ``--stop``, ``--skip`` and ``--module-only``
+  * add documentation on :ref:`manipulating_dependencies`, covering ``--filter-deps`` and ``--hide-deps``
+  * document ``-module-syntax`` configuration option (see :ref:`module_syntax`)
+  * add note on detection of unknown ``$EASYBUILD``-prefixed environment variables (see :ref:`configuration_env_vars`)
+  * mention support for prepending/appending to ``--robot-paths`` (see :ref:`robot_search_path_prepend_append`)
+  * update release notes for EasyBuild v2.1.0 (see :ref:`release_notes`)
+
+* **release 20150425.01** (`Apr 25th 2015`):
+
+  * add documentation on :ref:`using_external_modules`
+
+* **release 20150407.01** (`Apr 7th 2015`):
+
+  * add link to :ref:`unit_tests` page in dedicated section at :ref:`installation` page
+    (see :ref:`install_running_unit_tests`)
+  * clarify relation between ``--installpath``, ``--prefix``, ``-subdir-*`` and ``--installpath-*``
+    configuration options (see :ref:`installpath`)
+  * mention ``--show-default-configfiles`` command line option in relevant section
+    (see :ref:`default_configuration_files`)
+
+* **release 20150327.01** (`Mar 27th 2015`):
+
+  * documented deprecated functionality w.r.t. error reporting (see :ref:`depr_error_reporting`)
+* **release 20150316.01** (`Mar 16th 2015`):
+
+  * include list of EasyBuild repositories cloned by ``install-EasyBuild-develop.sh`` script
+    (see :ref:`install_latest_develop_using_script`)
+* **release 20150312.01** (`Mar 12th 2015`):
+
+  * enhance documentation w.r.t. template values in configuration files (see :ref:`configuration_file_templates_constants`)
+  * improve documentation on ``--robot`` and ``--robot-paths`` (see :ref:`controlling_robot_search_path`)
+* **release 20150310.01** (`Mar 10th 2015`):
+
+  * document peculiarties w.r.t. dependencies in combination with a ``dummy`` toolchain (see :ref:`dependency_specs`)
+  * document ``clean_gists.py`` script (see :ref:`clean_gists_script`)
+  * mention taking into account of proxy settings for downloading sources (see :ref:`common_easyconfig_param_sources`)
 * **release 20150306.03** (`Mar 6th 2015`): add release notes for EasyBuild v2.0.0 (see :ref:`release_notes`)
 * **release 20150306.02** (`Mar 6th 2015`):
 

docs/Concepts_and_Terminology.rst

@@ -113,8 +113,8 @@ Recent releases of EasyBuild include out-of-the-box toolchain support for:
 
 .. _dummy_toolchain:
 
-Dummy toolchain
-~~~~~~~~~~~~~~~
+``dummy`` toolchain
+~~~~~~~~~~~~~~~~~~~
 
 The ``dummy`` toolchain is a special case. It is an `empty` toolchain, i.e. a toolchain without any components,
 and corresponds to using the readily available compilers and libraries (e.g., the ones provided by the OS, or
@@ -153,3 +153,20 @@ Therefore, reproducing installations becomes trivial.
 
 EasyBuild version |version| includes support for over 511 different software packages,
 spread over 2800 easyconfig files describing distinct builds.
+
+.. _extensions:
+
+Extensions
+----------
+
+Some software packages support installing additional add-ons alongside the 'main' software, either in the same
+installation prefix, or in a separate location.
+
+In EasyBuild, we use the neutral term '**extensions**' to refer these add-ons.
+
+Well-known examples include:
+
+* Perl modules (http://www.cpan.org/modules/)
+* Python packages (https://pypi.python.org/pypi)
+* R libraries (http://cran.r-project.org/web/packages/)
+* Ruby gems (http://guides.rubygems.org/what-is-a-gem/)

docs/Deprecated-functionality.rst

@@ -34,11 +34,43 @@ For authors of easyconfig files:
 
 For developers of easyblocks:
 
-*(nothing yet)*
+* :ref:`depr_error_reporting`
 
 For EasyBuild framework developers:
 
-*(nothing yet)*
+* :ref:`depr_error_reporting`
+
+.. _depr_error_reporting:
+
+Report errors by raising ``EasyBuildError`` rather than using log methods
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Use of the** ``error()`` **and** ``exception()`` **log methods is deprecated.**
+
+* *deprecated since:* EasyBuild v2.1.0 (April'15)
+* *removed in:* EasyBuild v3.0
+* *alternative(s)*: **use** ``raise EasyBuildError(...)`` **instead**
+
+The ``error()`` and ``exception()`` log methods defined by EasyBuild (in the ``easybuild.tools.build_log`` module)
+do not match the semantics of the `standard Python log methods
+<https://docs.python.org/2/library/logging.html#logging.Logger.error>`_, in the sense that they also raise an
+exception next to logging messages.
+
+This may cause problems when 3rd party libraries (e.g., `gc3pie <https://pypi.python.org/pypi/gc3pie>`_) are being
+used by EasyBuild, since they may be using these log methods without expecting an exception being raised.
+
+The custom definitions for the ``error()`` and ``exception()`` log methods will be removed in EasyBuild v3.0.
+
+Hence, these log methods should no longer be used to report errors since they will not raise an exception anymore once
+their custom definitions are removed. Note that this applies both to the EasyBuild framework and to (custom) easyblocks.
+
+To report errors, an ``EasyBuildError`` should be raised instead. For example:
+
+.. code::
+
+    # make sure config.sh script is there
+    if not os.path.exists(os.path.join(self.builddir, 'config.sh')):
+        raise EasyBuildError("config.sh script is missing in %s", self.builddir)
 
 .. _deprecation_policy:
 

docs/Installation.rst

@@ -127,6 +127,8 @@ For example::
   may make a big difference, if you have installed both versions 1.9.0 and 1.15.2,
   with respect to what is the version being loaded by default.
 
+.. _install_running_unit_tests:
+
 Running unit tests
 ~~~~~~~~~~~~~~~~~~
 
@@ -135,6 +137,8 @@ After completion of the bootstrap procedure and loading the
 
     python -m test.framework.suite
 
+More details about the EasyBuild unit tests are available at :ref:`unit_tests`.
+
 If this does not complete successfully, `please open an issue`_ to report it.
 
 .. _please open an issue: https://github.com/hpcugent/easybuild-framework/issues/new

docs/Installation_Alternative.rst

@@ -174,7 +174,24 @@ Installation of latest development version using provided script
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 After you have forked each of the EasyBuild repositories on GitHub (+ vsc-base), you can set up a development version
-of EasyBuild, in the following manner::
+of EasyBuild using the ``install-EasyBuild-develop.sh`` script.
+
+This script will clone the different EasyBuild repositories from GitHub:
+
+ * `easybuild <https://github.com/hpcugent/easybuild>`_:
+   EasyBuild metapackage & documentation sources for http://easybuild.readthedocs.org
+ * `vsc-base <https://github.com/hpcugent/vsc-base>`_:
+   dependency for EasyBuild framework (logging, command line interface, ...)
+ * `easybuild-framework <https://github.com/hpcugent/easybuild-framework>`_:
+   EasyBuild framework
+ * `easybuild-easyblocks <https://github.com/hpcugent/easybuild-easyblocks>`_:
+   collection of easyblocks
+ * `easybuild-easyconfigs <https://github.com/hpcugent/easybuild-easyconfigs>`_:
+   collection of easyconfig files
+ * `easybuild-wiki <https://github.com/hpcugent/easybuild-wiki>`_:
+   EasyBuild wiki pages
+
+It can be used as follows::
 
     # pick an installation prefix (adjust as you like)
     INSTALL_PREFIX=$(mktemp -d $HOME/EasyBuild-XXXXXX)

docs/Manipulating_dependencies.rst

@@ -0,0 +1,75 @@
+.. _manipulating_dependencies:
+
+Manipulating dependencies
+=========================
+
+A couple of different ways are available to manipulate the list of dependencies that are specified for the software
+packages being installed.
+
+.. _filter_deps:
+
+Filtering out dependencies using ``--filter-deps``
+--------------------------------------------------
+
+To avoid that certain dependencies are being installed, a list of software names can be specified to ``--filter-deps``.
+Any time a dependency with a name from this list is specified, it will be simply filtered out by EasyBuild, and
+thus disregarded when resolving dependencies, loading modules for the dependencies in the build environment, and
+including '``module load``' statements in the generated module files.
+
+This can be useful when particular tools and libraries are already provided by OS packages (or in some other way),
+and should not be reinstalled as modules by EasyBuild.
+
+For example:
+
+* overview of dependencies of HDF5::
+
+    $ eb HDF5-1.8.13-intel-2015a.eb -D
+     ...
+     * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a)
+     * [ ] $CFGS/z/zlib/zlib-1.2.8-intel-2015a.eb (module: zlib/1.2.8-intel-2015a)   
+     * [ ] $CFGS/s/Szip/Szip-2.1-intel-2015a.eb (module: Szip/2.1-intel-2015a)
+     * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a)
+
+* overview of dependencies of HDF5, with zlib and Szip excluded::
+
+    $ eb HDF5-1.8.13-intel-2015a.eb --filter-deps=zlib,Szip -D 
+     ... 
+     * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a)
+     * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a)
+
+
+.. _hide_deps:
+
+Installing dependencies as hidden modules using ``--hide-deps``
+---------------------------------------------------------------
+
+Selected software packages can be marked to be installed as hidden modules (i.e., modules that are not visible via
+'``module avail``', but can be loaded) whenever they are included as a dependency, via the ``--hide-deps``
+configuration option.
+
+For example (note the preceding '``.``' in the last part of the module names for zlib and Szip)::
+
+ $ eb HDF5-1.8.13-intel-2015a.eb --hide-deps=zlib,Szip -D
+  ...
+  * [ ] $CFGS/i/intel/intel-2015a.eb (module: intel/2015a)
+  * [ ] $CFGS/z/zlib/zlib-1.2.8-intel-2015a.eb (module: zlib/.1.2.8-intel-2015a)
+  * [ ] $CFGS/s/Szip/Szip-2.1-intel-2015a.eb (module: Szip/.2.1-intel-2015a)
+  * [ ] $CFGS/h/HDF5/HDF5-1.8.13-intel-2015a.eb (module: HDF5/1.8.13-intel-2015a)
+
+.. note::
+  Using Lmod (version >= 5.7.5), hidden modules can be made visible in the output of '``module avail``' using the
+  ``--show-hidden`` option.
+
+  For example::
+
+   $ module avail bzip2
+
+   Use "module spider" to find all possible modules.
+   Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
+
+   $ module --show-hidden avail bzip2
+   ----- /home/example/.local/easybuild/modules/all -----
+   bzip2/.1.0.6
+
+   Use "module spider" to find all possible modules.
+   Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".