Skip to content

Commit

Permalink
Merge branch 'development' into 'master'
Browse files Browse the repository at this point in the history 
Several updates

See merge request brigaud/sw-lsst-eu!12
  • Loading branch information
Fabio Hernandez committed Dec 12, 2018
2 parents ac68782 + cba32e3 commit 2f5d7a0
Show file tree
Hide file tree
Showing 2 changed files with 70 additions and 52 deletions.
4 changes: 2 additions & 2 deletions docs/overview.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ With this cloud-based software distribution mechanism **you will be able to use

This service is built upon CERN's `CernVM file system <https://cvmfs.readthedocs.io>`_, a POSIX read-only file system in user space designed for large-scale software distribution. A software agent running on your computer interacts with servers via standard protocols, downloads the files on demand, caches them on your computer and exposes them via the operating system's file system interface.

You may want to browse the current contents of this software repository via `CERN's repository monitor <http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/sw.lsst.eu/latest>`_ before deciding if this service is suitable for your own needs.
You may want to **browse the current contents of this software repository** via `CERN's repository monitor <http://cernvm-monitor.cern.ch/cvmfs-monitor/cb/browser/sw.lsst.eu/latest>`_ before deciding if this service is suitable for your own needs.

In this section you will find information on the benefits of this mechanism, its intended audience, the platforms where it is supported and how to get started.

Expand Down Expand Up @@ -38,7 +38,7 @@ Intended Audience

This mechanism of distributing the LSST software, which supplements the `other mechanisms <https://pipelines.lsst.io/install/index.html>`_ provided by the project, is intended in particular for **individuals** willing to use the LSST software on their personnal computers.

However, it is not limited to that use case. The same mechanism is used for making the LSST software available to the computers in the CC-IN2P3's `login <http://doc.lsst.eu/ccin2p3/ccin2p3.html#login-farm>`_ and `batch <http://doc.lsst.eu/ccin2p3/ccin2p3.html#batch-farm>`_ farms. This is considered beneficial for reproducibility purposes, since as a scientist you can use exactly the same version of the LSST software on your personal computer, as well as in your notebooks and in your batch jobs executing at CC-IN2P3.
However, it is not limited to that use case. The same mechanism is used for making the LSST software available to the computers in the CC-IN2P3's `login <http://doc.lsst.eu/ccin2p3/ccin2p3.html#login-farm>`_ and `batch <http://doc.lsst.eu/ccin2p3/ccin2p3.html#batch-farm>`_ farms. This is considered beneficial for reproducibility purposes, since as a scientist you can use exactly the same version of the LSST software on your personal computer, as well as in your notebooks and in your batch or grid jobs executing at CC-IN2P3, `NERSC <https://www.nersc.gov>`_ and `OpenScienceGrid <https://opensciencegrid.org>`_ sites.


===================
Expand Down
118 changes: 68 additions & 50 deletions docs/usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,55 +11,73 @@ In this section you will find information on how to use the online LSST software
Repository Layout
=================

The namespace under ``/cvmfs/sw.lsst.eu`` is meant to be self-explanatory. There you will find a sub-directory per supported platform (i.e. ``darwin-x86_64``, ``linux-x86_64``), a subdirectory for each distribution (e.g. ``lsst_distrib``) and a subdirectory for each available release (e.g. ``v15.0``, ``w_2018_19``). It looks like:
The namespace under ``/cvmfs/sw.lsst.eu`` is meant to be self-explanatory. There you will find a sub-directory per supported platform (i.e. ``darwin-x86_64``, ``linux-x86_64``), a subdirectory for each distribution (e.g. ``lsst_distrib``, ``lsst_sims``) and a subdirectory for each available release (e.g. ``v16.0``, ``w_2018_48``). It looks like:

.. code-block:: bash
$ tree -L 3 /cvmfs/sw.lsst.eu
/cvmfs/sw.lsst.eu
|-- darwin-x86_64
| `-- lsst_distrib
| |-- v15.0
| |-- v16.0
| |-- w_2018_14
| |-- w_2018_15
| |-- w_2018_16
| |-- w_2018_17
| |-- w_2018_18
| |-- w_2018_19
| |-- w_2018_20
| |-- w_2018_21
| |-- w_2018_22
| |-- w_2018_23
| |-- w_2018_24
| `-- w_2018_25
`-- linux-x86_64
|-- lsst_distrib
| |-- v15.0
| |-- v16.0
| |-- w_2018_14
| |-- w_2018_15
| |-- w_2018_16
| |-- w_2018_17
| |-- w_2018_18
| |-- w_2018_19
| |-- w_2018_20
| |-- w_2018_21
| |-- w_2018_22
| |-- w_2018_23
| |-- w_2018_24
| `-- w_2018_25
`-- lsst_sims
`-- sims_2_8_0
34 directories, 0 files
Names of directories containing **stable releases** start with letter "v" (e.g. ``v15.0``) and directories where **weekly releases** are located are named starting with letter "w" (e.g. ``w_2018_19``). ``lsst_distrib`` is the name of the LSST distribution, that is, a coherent set of packages that together form the LSST science pipelines. Each release of the LSST software is built specifically for delivery via CernVM-FS according to the `official instructions <https://pipelines.lsst.io>`_.

Each release of the LSST software you will find under ``/cvmfs/sw.lsst.eu``, be it stable or weekly, is mostly self contained: it includes its own EUPS (see below), its own **Python 3** distribution (typically `miniconda <https://www.anaconda.com/download>`_) and its own set of external packages that specific release depends on (e.g. ``numpy``, ``cfitsio``, etc.). In particular, since the Python distribution installed with each release includes its own interpreter, each release is independent and configured so **it does not conflict with other Python interpreter** you may have already installed on your computer.
├── darwin-x86_64
│   └── lsst_distrib
│   ├── v15.0
│   ├── v16.0
│   ├── w_2018_30
│   ├── w_2018_31
│   ├── w_2018_32
│   ├── w_2018_33
│   ├── w_2018_34
│   ├── w_2018_35
│   ├── w_2018_36
│   ├── w_2018_37
│   ├── w_2018_38
│   ├── w_2018_39
│   ├── w_2018_40
│   ├── w_2018_41
│   ├── w_2018_42
│   ├── w_2018_43
│   ├── w_2018_44
│   ├── w_2018_45
│   ├── w_2018_46
│   ├── w_2018_47
│   ├── w_2018_48
│   └── w_2018_49
└── linux-x86_64
├── lsst_distrib
│   ├── v15.0
│   ├── v16.0
│   ├── w_2018_30
│   ├── w_2018_31
│   ├── w_2018_32
│   ├── w_2018_33
│   ├── w_2018_34
│   ├── w_2018_35
│   ├── w_2018_36
│   ├── w_2018_37
│   ├── w_2018_38
│   ├── w_2018_39
│   ├── w_2018_40
│   ├── w_2018_41
│   ├── w_2018_42
│   ├── w_2018_43
│   ├── w_2018_44
│   ├── w_2018_45
│   ├── w_2018_46
│   ├── w_2018_47
│   ├── w_2018_48
│   └── w_2018_49
└── lsst_sims
├── sims_2_10_0
├── sims_2_13_1
├── sims_2_8_0
├── sims_2_9_0
└── sims_w_2018_49
54 directories, 0 files
Names of directories where **weekly releases** are located are named starting with letter "w" (e.g. ``w_2018_48``) or "sims_w" (e.g. ``sims_w_2018_49``). **Stable releases** of ``lsst_distrib`` start with letter "v" (e.g. ``v16.0``) and stable releases of ``lsst_sims`` are named like ``sims_2_13_1``. ``lsst_distrib`` is the name of the LSST distribution, that is, a coherent set of packages that together form the LSST science pipelines. Each release of the software is built from sources, specifically for delivery via CernVM-FS according to the `official instructions <https://pipelines.lsst.io>`_.

Each release of the software you will find under ``/cvmfs/sw.lsst.eu``, be it stable or weekly, is mostly self contained: it includes its own EUPS (see below), its own **Python 3** distribution (typically `miniconda <https://www.anaconda.com/download>`_) and its own set of external packages that specific release depends on (e.g. ``numpy``, ``cfitsio``, etc.). In particular, since the Python distribution installed with each release includes its own interpreter, each release is independent and configured so **it does not conflict with other Python interpreter** you may have already installed on your computer.

.. important::

Expand All @@ -72,12 +90,12 @@ Each release of the LSST software you will find under ``/cvmfs/sw.lsst.eu``, be
Basic Usage
===========

The first step for using the LSST science pipelines is to select the release you want to use and bootstrap your environment for that specific release. For instance, to use LSST ``v15.0`` on a Linux computer do:
The first step for using the LSST science pipelines is to select the release you want to use and bootstrap your environment for that specific release. For instance, to use LSST ``v16.0`` on a Linux computer do:

.. code-block:: bash
# Open a new terminal session using a BASH shell
$ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v15.0/loadLSST.bash
$ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v16.0/loadLSST.bash
As a result of executing this command, some environmental variables are extended or initialized, such as ``PATH``, ``PYTHONPATH``, ``LD_LIBRARY_PATH`` and ``EUPS_PATH``.

Expand Down Expand Up @@ -110,14 +128,14 @@ After these steps, your working environment is modified so that you can use the
...
If later on you need to work with a different release, say weekly ``w_2018_19``, **you must create a new terminal session** and configure your environment for the that specific release. For instance:
If later on you need to work with a different release, say weekly ``w_2018_48``, **you must create a new terminal session** and configure your environment for the that specific release. For instance:

.. code-block:: bash
# In a new terminal session with BASH shell
$ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2018_19/loadLSST.bash
$ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2018_48/loadLSST.bash
# From this point on, your environment is set up to use release w_2018_19
# From this point on, your environment is set up to use release w_2018_48 of lsst_distrib
At this point you may want to `run the LSST demo <https://pipelines.lsst.io/install/demo.html#download-the-demo-project>`_ and read the tutorials on `how to use the LSST Science Pipelines <https://pipelines.lsst.io/getting-started/index.html#getting-started-tutorials>`_.

Expand Down Expand Up @@ -215,7 +233,7 @@ Let's suppose that you want to use your own version of one of the products inclu
# and verify that now EUPS knows about your private version
$ eups declare -r $HOME/obs_cfht obs_cfht my_private_obs_cfht
$ eups list obs_cfht
15.0-5-g891f9b3 w_latest w_2018_25 current setup
15.0-5-g891f9b3 w_latest w_2018_48 current setup
my_private_obs_cfht
# In order to use your private version you need to set it up first
Expand Down

0 comments on commit 2f5d7a0

Please sign in to comment.