diff --git a/docs/source/conf.py b/docs/source/conf.py index 2461e7c..ce9d659 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -84,5 +84,6 @@ # autodoc_pydantic_model_show_validator_summary = False autodoc_pydantic_model_member_order = "bysource" intersphinx_mapping = { - "dgbowl_schemas": ("https://dgbowl.github.io/dgbowl-schemas/main", None) + "dgbowl_schemas": ("https://dgbowl.github.io/dgbowl-schemas/main", None), + "xarray": ("https://docs.xarray.dev/en/stable", None), } diff --git a/docs/source/driver_develop.rst b/docs/source/driver_develop.rst new file mode 100644 index 0000000..5b5ccb3 --- /dev/null +++ b/docs/source/driver_develop.rst @@ -0,0 +1,55 @@ +.. _driver_develop: + +Developing **tomato** drivers +----------------------------- +As of ``tomato-1.0``, all device *drivers* are developed as separate Python packages with their own documentation and versioning. To ensure compatibility of the Manager between the ``tomato-driver`` process and the implementation of the *driver*, an abstract class :class:`~tomato.driverManager_1_0.ModelInterface` is provided. A class inheriting from this abstract class, with the name :class:`DriverManager`, **has** to be available when the selected *driver* module is imported. + +.. note:: + + The :class:`~tomato.driverManager_1_0.ModelInteface` is versioned. Your driver should target a single version of this Manager by inheriting from only one such abstract class. **Any deprecation notices will be provided well in advance directly to driver maintainers.** Support for :mod:`~tomato.driverManager_1_0` introduced in ``tomato-1.0`` is guaranteed until at least ``tomato-3.0``. + +Bootstrapping a *driver* process +```````````````````````````````` +When the *driver* process is launched (as a ``tomato-driver``), it's given information about how to connect to the ``tomato-daemon`` process and which device *driver* to spawn. Once a connection to the ``tomato-daemon`` is established, the *driver* settings are fetched, and the :class:`DriverManager` is instantiated passing any settings to the constructor. Then, all *components* on all *devices* of this *driver* type that are known to ``tomato-daemon`` are registered using the :func:`dev_register` function. + +.. note:: + + Each *driver* creates a separate log file for each port **tomato** has been executed with. The logfile is stored in the same location as the ``tomato-daemon`` logs, i.e. as configured under the ``jobs.storage`` option. The verbosity of the ``tomato-driver`` process is inherited from the ``tomato-daemon`` process. + +Communication between *jobs* and *drivers* +`````````````````````````````````````````` +After the *driver* process is bootstrapped, it enters the main loop, listening for commands to action or pass to the :class:`ModelInterface`. Therefore, if a *job* needs to submit a :class:`Task`, it passes the :class:`Task` to the ``tomato-driver`` process, which actions it on the appropriate *component* using the :func:`task_submit` function. Similarly, if a *job* decides to poll the *driver* for data, it does so using the :func:`task_data` function. + +In general, methods of the :class:`ModelInterface` that are prefixed with ``dev`` deal with managing *devices* or their *components* on the *driver*, methods prefixed with ``task`` deal with managing :class:`Tasks` running or submitted to *components*, and methods without a prefix deal with configuration or status of the *driver* itself. + +.. note:: + + The :class:`ModelInterface` contains a sub-class :class:`DriverManager`. In general, the :class:`ModelInterface` acts as a pass-through to the (abstract) methods of the :class:`DriverManager`; e.g. :func:`ModelInterface.dev_get_attr` is a passthrough function to the appropriate :func:`DriverManager.get_attr`. + + We expect most of the work in implementing a new *driver* will actually take place in the :class:`DriverManager` class. + +Currently, when a :class:`Task` is submitted to a *component*, a new :class:`Thread` is launched on that *component* that takes care of preparing the *component* (via :func:`DriverManager.prepare_task`), executing the :class:`Task` (via :func:`DriverManager.task_runner`), and periodically polling the hardware for data (via the abstract :func:`DriverManager.do_task`). As each *component* can only run one :class:`Task` at the same time, subsequently submitted :class:`Tasks` are stored in a ``task_list``, which is a :class:`Queue` used to communicate with the worker :class:`Thread`. This worker :class:`Thread` is reinstantiated at the end of every :class:`Task`. + +.. note:: + + Direct access to the :class:`DriverManager.data` object is not thread-safe. Therefore, a reentrant lock (:class:`RLock`) object is provided as :class:`DriverManager.datalock`. Reading or writing to the :obj:`DriverManager.data` with the exception of the :func:`get_data` and :func:`do_task` methods should be only carried out when this :obj:`datalock` is acquired, e.g. using a context manager. + +.. note:: + + The :class:`DriverManager.data` object is intended to cache data between :func:`get_data` calls initiated by the *job*. This object is therefore cleared whenever :func:`get_data` is called; it is the responsibility of the ``tomato-job`` process to append or store any new data. + + To access the status of the *component*, the :class:`DriverManager` provides a :func:`status` method. The implementation of what is reported as *component* status (including e.g. returning latest cached datapoint) is up to the developers of each *driver*. + +Best Practices when developing a *driver* +````````````````````````````````````````` +- We follow the usual Python (PEP-8) convention of ``_``-prefixed methods and attributes considered to be private. However, there is no way to enforce such privacy in Python. +- The :func:`DriverManager.attrs` defines the variable attributes of the *component* that should be accessible, using :class:`Attr`. All entries in :func:`attrs` should be present in :obj:`DriverManager.data`. There should be no entries in :obj:`data` that are not in returned by :func:`attrs`. +- Each :class:`DriverManager` contains a link to its parent :class:`ModelInterface` in the :obj:`DriverManager.driver` object. +- Internal functions of the :class:`DriverManager` and :class:`ModelInterface` should be re-used wherever possible. E.g., reading *component* attributes should always be carried out using :func:`get_attr`. + +ModelInterface ver. 1.0 +``````````````````````` + +.. autoclass:: tomato.driverinterface_1_0.ModelInterface + :no-index: + :members: diff --git a/docs/source/index.rst b/docs/source/index.rst index 17ecceb..8fb1302 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,14 +18,20 @@ - the job and job queue configuration utility, :mod:`tomato.ketchup`; - a set of device drivers, see the :ref:`driver library`. -This project has received funding from the European Union’s Horizon 2020 research -and innovation programme under grant agreement No 957189. The project is part of -BATTERY 2030+, the large-scale European research initiative for inventing the -sustainable batteries of the future. +This project has received funding from: + +- the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 957189, +- the Emmy Noether Programme of the DFG, under project No. 490703766 +- the Priority Programme 2419 HyCAM of the DFG, under project No. 523873650 + +The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future. .. codeauthor:: Peter Kraus, - Loris Ercole + Alexandre Gbocho, + Graham Kimbell, + Loris Ercole, + .. toctree:: :maxdepth: 1 @@ -34,6 +40,7 @@ sustainable batteries of the future. installation quickstart usage + driver_develop version .. _driver library: @@ -42,6 +49,10 @@ sustainable batteries of the future. :maxdepth: 1 :caption: tomato driver library + tomato-example-counter + tomato-psutil + tomato-bronkhorst + tomato-jumo .. toctree:: :maxdepth: 1 diff --git a/docs/source/installation.rst b/docs/source/installation.rst index f95ba48..51cbc27 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -2,26 +2,21 @@ Installation ------------ -Pre-built wheels of **tomato** are available on `PyPI `_ -and can be installed using: +Pre-built wheels of **tomato** are available on `PyPI `_ and can be installed using: .. code:: pip install tomato[docs,testing] -We strongly recommend installing **tomato** into a separate ``conda`` or ``venv`` -environment. +We strongly recommend installing **tomato** into a separate ``conda`` or ``venv`` environment. .. note:: - The optional targets ``[docs]`` and ``[testing]`` will install packages required - for building this documentation and running the test-suite, respectively. + The optional targets ``[docs]`` and ``[testing]`` will install packages required for building this documentation and running the test-suite, respectively. Testing the installation ```````````````````````` -To run the test-suite, you need to first install **tomato** using the above command, -and then you need to clone the git repository, and launch ``pytest`` from within the -created ``tomato`` folder: +To run the test-suite, you need to first install **tomato** using the above command, and then you need to clone the git repository, and launch ``pytest`` from within the created ``tomato`` folder: .. code:: diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index 9367082..0cba0bb 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -10,9 +10,7 @@ First time set-up This section assumes that **tomato** has been successfully :ref:`installed`. -To prepare **tomato** for its first execution, a *settings file* containing the basic -configuration of the driver needs to be placed in **tomato's** *appdir* folder. By -default, the *appdir* path is: +To prepare **tomato** before its first run, a *settings file* containing the basic configuration for the :mod:`tomato.daemon` needs to be placed in **tomato's** *appdir* folder. By default, the *appdir* path is: - ``$env:localappdata\dgbowl\tomato\`` on Windows, - ``$HOME/.config/tomato/`` on Linux. @@ -26,20 +24,17 @@ The easiest way to do create this file is using the provided ``tomato init`` com kraus@dorje:/home/kraus/$ tomato init Success: wrote default settings into /home/kraus/.config/tomato/1.0a1/settings.toml -Where *appdir* is ``/home/kraus/.config/tomato/1.0a1/``. The *appdir* can be specified -using the ``--appdir`` argument to **tomato**. +Where *appdir* is ``/home/kraus/.config/tomato/1.0a1/``. A custom *appdir* can be specified using the ``--appdir`` argument to **tomato**. **tomato** program flowchart ```````````````````````````` **tomato** currently contains two command line user-facing utilities/executables: -- :mod:`~tomato.tomato`, responsible for management of the state daemon process, and +- :mod:`~tomato.tomato`, responsible for management of the daemon process, and - :mod:`~tomato.ketchup`, responsible for job submission and management. -These two executables then internally spawn other processes, including the state daemon -``tomato-daemon``, a ``tomato-job`` process for every running job, and a ``tomato-driver`` -process for managing all devices of a certain driver type. +These two executables then internally spawn other processes, including the state daemon process ``tomato-daemon``, a ``tomato-job`` process for every running job, and a ``tomato-driver`` process for each driver type, managing all devices of a that type. .. mermaid:: @@ -75,11 +70,11 @@ process for managing all devices of a certain driver type. ````````````````````````````` The following concepts are used in **tomato**: -- *devices*, which represent separately addressable and optionally multichannel instruments in the lab, +- *devices*, which represent separately addressable (and optionally multichannel) instruments in the lab, - *components*, which represent the individual channels of each *device*, describing the role of this *component* and its associated capabilities, -- *pipelines*, which represent the real world organisation of device *components* into independent experimental set-ups, optionally containing *components* from one or more *device*, -- *jobs*, which are processes that carry out a *payload* (i.e. a set of experimental instructions) on a *pipeline*, and -- *drivers*, which are processes managing all *devices* of a certain driver type and their *component*: +- *pipelines*, which represent the real world organisation of device *components* into independent experimental set-ups, containing *components* from one or more *device*, +- *jobs*, which are processes that carry out a *payload* (i.e. a set of experimental instructions) on a *pipeline* by instructing its *components* to perform individual *tasks*, and +- *drivers*, which are separate processes, each managing all defined *devices* of a certain driver type as well as their *components*: .. mermaid:: @@ -147,9 +142,7 @@ The following concepts are used in **tomato**: Settings file ````````````` -The *settings file* contains the basic information required to start the ``tomato-daemon``. -This is a ``settings.toml`` file, which has to be located in the *appdir*. The default -file generated by ``tomato init`` looks similar to the below example: +The *settings file* contains the basic information required to start the ``tomato-daemon``. The filename is hard-coded to be a ``settings.toml`` file, which has to be located in the *appdir*. The default file generated by ``tomato init`` looks similar to the below example: .. code-block:: toml :linenos: @@ -166,32 +159,25 @@ file generated by ``tomato init`` looks similar to the below example: example_counter.testpar = 1234 -In addition to the *appdir*, a second path, *datadir*, is used to specify the location of -the data created by **tomato**. The default *datadir* is: +In addition to the *appdir*, a second path, *datadir*, is used to specify the location of the data created by **tomato**. The default *datadir* is: - ``$env:localappdata\dgbowl\tomato\`` on Windows, - ``$HOME/.local/share/tomato/`` on Linux. In the default *settings file* shown above, the following entries are specified: -- ``jobs.storage`` which is the directory where the data and logs of **tomato** jobs - will be stored, -- ``devices.config`` which points to a ``yaml``-formatted :ref:`devices file `, - defining the hardware configuration of the devices managed by **tomato**. +- ``jobs.storage`` which is the directory where the data and logs of **tomato** jobs will be stored, +- ``devices.config`` which points to a ``yaml``-formatted :ref:`devices file `, defining the hardware configuration of the devices managed by **tomato**. -Additional, *driver*-specific settings may be provided in the ``[drivers]`` section, -following the example of the ``drivers.example_counter.testpar`` entry. +Additional, *driver*-specific settings may be provided in the ``[drivers]`` section, following the example of the ``drivers.example_counter.testpar`` entry. These *driver*-specific settings are passed to each *driver* when its process is launched and the :class:`DriverInterface` is initialised, and can therefore contain paths to various libraries or other files necessary for the *driver* to function. .. _devfile: Devices file ```````````` -This ``yaml``-formatted file contains information about each *device*, corresponding to -an individual piece of hardware managed by **tomato**, as well as information about the -organisation of such *devices* into *pipelines*. +This ``yaml``-formatted file contains information about each *device*, corresponding to an individual piece of hardware managed by **tomato**, as well as information about the organisation of the individually-addressable *components* of those *devices* into *pipelines*. -When the devices file is not present, the default file shipped with **tomato** will be -used instead: +When the devices file is not present, the default file shipped with **tomato** will be used instead: **devices** section ******************* @@ -207,12 +193,9 @@ The ``devices`` section of the default *devices file* is shown below: channels: ["1"] pollrate: 1 -Here, we define a single device using the :mod:`~tomato.drivers.example_counter` driver. -The definition includes the ``address`` of the device (:class:`str` type) as well as an -enumeration of individually-addressable channels the device has (:class:`list[str]`). +Here, we define a single device using the :mod:`~tomato.drivers.example_counter` driver. The definition includes the ``address`` of the device (:class:`str` type) as well as an enumeration of individually-addressable channels the device has (:class:`list[str]`). -For example, the devices shown in the :ref:`concepts flowchart ` above would -be defined as: +For example, the devices shown in the :ref:`concepts flowchart ` above would be defined as: .. code-block:: yaml :linenos: @@ -237,9 +220,7 @@ be defined as: .. note:: - The ``pollrate`` here is an internal setting for **tomato**. It is the frequency - with which the job interacting with the device driver asks the driver for data, - **not** the frequency with which the driver asks the hardware for data! + The ``pollrate`` here is an internal setting for **tomato**. It is the frequency with which the *job* interacting with the device *driver* asks the *driver* for data, **not** the frequency with which the *driver* asks the hardware for data! This latter "sampling" frequency can be defined in each individual *task* submitted as a *payload* of a *job*. **pipelines** section ********************* @@ -256,10 +237,7 @@ The default ``pipelines`` section looks as follows: device: dev-counter channel: "1" -Here, a single *pipeline* called ``pip-counter`` is defined to contain the one available -channel of the ``dev-counter`` device (defined on line 5) shown further above. For multi -channel devices, it is also possible to define a *pipeline* for each channel automatically. -For example, with the following definition: +Here, a single *pipeline* called ``pip-counter`` is defined to contain the one available channel of the ``dev-counter`` device (defined on line 5) shown further above. For multi channel devices, it is also possible to define a *pipeline* for each channel automatically, e.g. using the following definition: .. code-block:: yaml :linenos: @@ -272,13 +250,9 @@ For example, with the following definition: device: dev-counter channel: each -a set of pipelines would be created using each of the available channels in -``dev-counter``, replacing the ``*`` in line 2 using the channel. Therefore, one -*pipeline* with name ``pip-counter-1`` would be created. However, if the ``dev-counter`` -contained more than one channel, a *pipeline* would be generated for each channel. +Here, a set of pipelines would be created using each of the available channels in ``dev-counter``, replacing the ``*`` in line 2 using the channel. Therefore, one *pipeline* with name ``pip-counter-1`` would be created. However, if the ``dev-counter`` were to contain more than one channel, a *pipeline* would be generated for each channel. -Finally, a definition for the pipelines shown in the :ref:`concepts flowchart ` -above can be defined as: +Finally, a definition for the pipelines shown in the :ref:`concepts flowchart ` above can be defined as: .. code-block:: yaml :linenos: @@ -310,8 +284,19 @@ above can be defined as: Payload file ```````````` -The *payload* file contains all information required to enter a *job* into the queue and -allow its assignment onto a *pipeline*. The overall schema of the *payload* is defined -in the :mod:`dgbowl_schemas.tomato` module, and is parsed using :func:`dgbowl_schemas.tomato.to_payload`: +The *payload* file contains all information required to enter a *job* into the queue and allow its assignment onto a *pipeline*. The overall schema of the *payload* is defined in the :mod:`dgbowl_schemas.tomato` module, and is parsed using :func:`dgbowl_schemas.tomato.to_payload`: -.. autopydantic_model:: dgbowl_schemas.tomato.payload_1_0.Payload \ No newline at end of file +.. autopydantic_model:: dgbowl_schemas.tomato.payload.Payload + + +.. note:: + + Of particular importance here is the specification of the individual :class:`Tasks`. Some general :class:`Task` parameters are abstracted by **tomato**, such as the ``max_duration``, ``sampling_interval`` and ``polling_interval``. The other are *driver* or *component* specific, and can be specified using the ``technique_params`` :class:`dict`. + +.. autopydantic_model:: dgbowl_schemas.tomato.payload.Task + +.. warning:: + + Currently, **tomato** performs no validation of any ``technique_params`` at *job* submission. It also does not check whether the ``sampling_interval`` is realistic. This means your *jobs* may crash when the ``technique_params`` are out of bounds when the :class:`Task` is being executed. + + However, such validation is planned for ``tomato-2.0``, see https://github.com/dgbowl/tomato/issues/99. \ No newline at end of file diff --git a/docs/source/usage.rst b/docs/source/usage.rst index 40d1980..aaefda6 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -11,34 +11,25 @@ Starting :mod:`tomato.daemon` .. note:: - For instructions on how to set **tomato** up for a first run, see the - :ref:`quickstart`. + For instructions on how to set **tomato** up for a first run, see the :ref:`quickstart`. -Provided a :ref:`settings file ` exists, the job scheduler ``tomato`` can be -started on the default *port* using: +Provided a :ref:`settings file ` exists, the daemon process ``tomato-daemon`` can be started on the default *port* using: .. code:: bash tomato start -The daemon keeps track of *pipelines* configured in the :ref:`device file `, -and schedules *jobs* from the queue onto them. See the :ref:`concepts flowchart ` -for a more detailed overview. +The daemon keeps track of *pipelines* configured in the :ref:`device file `, and schedules *jobs* from the queue onto them. See the :ref:`concepts flowchart ` for a more detailed overview. .. note:: - Multiple instances of the :mod:`tomato.daemon` can be running at a single PC, - provided a different *port* is specified using ``--port`` argument to ``tomato`` - and ``ketchup``. + Multiple instances of the :mod:`tomato.daemon` can be running at a single PC, provided a different *port* is specified using ``--port`` argument to ``tomato`` and ``ketchup``. Using :mod:`~tomato.tomato` ``````````````````````````` -The :mod:`tomato.tomato` executable is used to configure, start, and manage the -**tomato** daemon, as well as load / eject samples to / from *pipelines* and mark them -ready. +The :mod:`tomato.tomato` executable is used to configure, start, and manage the **tomato** daemon, as well as load / eject samples to / from *pipelines* and mark them ready. - #. **To configure** the **tomato** daemon by creating a default - :ref:`settings file `, run: + #. **To configure** the **tomato** daemon by creating a default :ref:`settings file `, run: .. code-block:: bash @@ -50,9 +41,7 @@ ready. >>> tomato start - This will read the :ref:`settings file `, and parse the - :ref:`device file ` listed within. To start the **daemon on an - alternative port**, run: + This will read the :ref:`settings file `, and parse the :ref:`device file ` listed within. To start the **daemon on an alternative port**, run: .. code-block:: bash @@ -60,9 +49,7 @@ ready. .. warning:: - All ``tomato`` and ``ketchup`` commands intended to interact with the - **tomato** daemon running on an alternative port will have to be executed - with the same ``--port `` argument. + All ``tomato`` and ``ketchup`` commands intended to interact with the **tomato** daemon running on an alternative port will have to be executed with the same ``--port `` argument. #. **To stop** the **tomato** daemon, run: @@ -70,9 +57,9 @@ ready. >>> tomato stop - The daemon will only stop if there are no running jobs. However, a snapshot - of the daemon state will be generated. There is currently no way to stop the - **tomato** daemon cleanly while jobs are running. + .. note:: + + The daemon will only stop if there are no running jobs. However, a snapshot of the daemon state will be generated in any case. There is currently no clean way to stop the **tomato** daemon cleanly while jobs are running. #. **To reload settings** of a running **tomato** daemon, run: @@ -80,12 +67,13 @@ ready. >>> tomato reload - Currently, reloading *driver* settings from the :ref:`settings file ` - and adding *pipelines* and/or *devices* from the :ref:`device file ` is - supported. + Currently, reloading *driver* settings from the :ref:`settings file ` and managing *pipelines* and/or *devices* from the :ref:`device file ` is supported. Any *component* not present in a *pipeline* is automatically removed. + + .. note:: - #. **To manage individual pipelines** of a running **tomato** daemon, the following - commands are available: + The daemon will only remove *pipelines*, *devices* and *components* if they are not used by any running *job*. + + #. **To manage individual pipelines** of a running **tomato** daemon, the following commands are available: - For loading a sample into a *pipeline*: @@ -101,10 +89,11 @@ ready. >>> tomato pipeline eject - This will also succeed if the *pipeline* was already empty. It will fail - if the *pipeline* has a job running. + This will also succeed if the *pipeline* was already empty. It will fail if the *pipeline* has a job running. + + .. note:: - Ejecting a sample from any *pipeline* will mark the *pipeline* as not ready. + As a precaution, ejecting a sample from any *pipeline* will always mark the *pipeline* as not ready. - To mark a *pipeline* as ready: @@ -118,8 +107,7 @@ ready. Using :mod:`~tomato.ketchup` ```````````````````````````` -The :mod:`tomato.ketchup` executable is used to submit *payloads* to the daemon, and -to check the status of and to cancel *jobs* in the queue. +The :mod:`tomato.ketchup` executable is used to submit *payloads* to the daemon, and to check the status of and to cancel *jobs* in the queue. #. **To submit** a *job* using a *payload* contained in a :ref:`payfile`, run: @@ -131,8 +119,7 @@ to check the status of and to cancel *jobs* in the queue. .. note:: - For more information about how *jobs* are matched against *pipelines*, see the - documentation of the :mod:`~tomato.daemon` module. + For more information about how *jobs* are matched against *pipelines*, see the documentation of the :mod:`~tomato.daemon` module. #. **To check the status** of one or several *jobs* with known ``jobids``, run: @@ -140,8 +127,7 @@ to check the status of and to cancel *jobs* in the queue. >>> ketchup status - When executed without argument, the status of the whole queue will be returned. - The list of possible *job* statuses is: + When executed without argument, the status of the whole queue will be returned. The list of possible *job* statuses is: ======== =========================================================== Status Meaning @@ -161,61 +147,55 @@ to check the status of and to cancel *jobs* in the queue. >>> ketchup cancel - This will mark the *jobs* for cancellation by setting their status to ``rd``. - The :mod:`tomato.daemon` will then proceed with cancelling each *job*. + This will mark the *jobs* for cancellation by setting their status to ``rd``. The :mod:`tomato.daemon` will then proceed with cancelling each *job*. -*Jobs* submitted to the queue will remain in the queue until a *pipeline* meets all -of the following criteria: +*Jobs* submitted to the queue will remain in the queue until a *pipeline* meets all of the following criteria: - - A *pipeline* where all of the ``tasks`` specified in the *payload* are matched - by its ``capabilities`` must exist. Once the :mod:`tomato.daemon` finds such a - *pipeline*, the status of the *job* will change to ``qw``. - - The matching *pipeline* must contain a *sample* with a ``samplename`` that matches - the name specified in the *payload*. + - A *pipeline* where all of the ``technique_names`` specified in all :class:`Tasks` within the *payload* are matched by its ``capabilities`` must exist. Once the :mod:`tomato.daemon` finds such a *pipeline*, the status of the *job* will change to ``qw`` to indicate a suitable *pipeline* exists. + - The matching *pipeline* must contain a *sample* with a ``samplename`` that matches the name specified in the *payload*. - The matching *pipeline* must be marked as ``ready``. .. note:: - Further information about :mod:`~tomato.ketchup` is available in the documentation - of the :mod:`~tomato.ketchup` module. + Further information about :mod:`~tomato.ketchup` is available in the documentation of the :mod:`~tomato.ketchup` module. + +Machine-readable output +``````````````````````` +As of ``tomato-1.0``, the outputs of :mod:`~tomato.tomato` and :mod:`~tomato.ketchup` utilities can be requested as a yaml-formatted text, by passing the ``--yaml`` (or ``-y``) command-line parameter to the executables. Accessing output data ````````````````````` -Each *job* stores its data and logs in its own *job* folder, which is a subfolder of the -``jobs.storage`` folder specified in the :ref:`settings file `. +Each *job* stores its data and logs in its own *job* folder, which is a subfolder of the ``jobs.storage`` folder specified in the :ref:`settings file `. .. warning:: - While "live" *job* data is available in the *job* folder in pickled form, accessing - those files directly is not supported and may lead to race conditions and crashes. - If you need an up-to-date data archive, request a :ref:`snapshot `. - If you need the current status of a *device*, probe the responsible driver process. + While "live" *job* data is available in the *job* folder in pickled form, accessing those files directly is not supported and may lead to race conditions and crashes. If you need an up-to-date data archive, request a :ref:`snapshot `. If you need the current status of a *device*, probe the responsible driver process. + + Note that a *pipeline* dashboard functionality is planned for a future version of ``tomato``. Final job data ************** -By default, all data in the *job* folder is processed to create a NetCDF file. -a *datagram*, and zipped into a zip archive. This zip archive includes all raw -data files, the log file of the **tomato** job, and a copy of the full *payload* -in a ``json`` file. The *datagram* contains timestamped, unit-annotated raw data, -and includes instrumental uncertainties. +By default, all data in the *job* folder is processed to create a NetCDF file. The NetCDF files can be read using :func:`xaray.open_datatree`, returning a :class:`xarray.DataTree`. + +In the root node of the :class:`~xarray.DataTree`, a copy of the full *payload* is included, serialised as a json :class:`str`. Additionally, execution-specific metadata, such as the *pipeline* ``name``, and *job* submission/execution/completion time are stored on the root node, too. + +The child nodes of the :class:`~xarray.DataTree` contain the actual data from each *pipeline* *component*, unit-annotated using the CF Metadata Conventions. The node names correspond to the ``role`` that *component* fullfils in a *pipeline*. + +.. note:: + + As opposed to ``tomato-0.2``, in ``tomato-1.0`` we currently do not output measurement uncertainties. -Unless specified within the *payload*, the default location where these output files -will be placed is the ``cwd()`` where the ``ketchup submit`` command was executed; -the default filenames of the returned files are ``results..[zip,json]``. +Unless specified within the *payload*, the default location where these output files will be placed is the ``cwd()`` where the ``ketchup submit`` command was executed; the default filenames of the returned files are ``results..[zip,json]``. .. _snapshotting: Data snapshotting ***************** -While the *job* is running, access to an up-to-date snapshot of the data is provided -by :mod:`~tomato.ketchup`: +While the *job* is running, access to an up-to-date snapshot of the data is provided by :mod:`~tomato.ketchup`: .. code:: bash >>> ketchup snapshot -This will create an up-to-date ``snapshot..nc`` file in the current working dir. -The files are overwritten on subsequent invocations of ``ketchup snapshot``. An -automated, periodic snapshotting can be further configured within the *payload* -of the *job*. \ No newline at end of file +This will create an up-to-date ``snapshot..nc`` file in the current working dir. The files are overwritten on subsequent invocations of ``ketchup snapshot``. An automated, periodic snapshotting stored in a custom location can be further configured within the *payload* of the *job*. \ No newline at end of file diff --git a/docs/source/version.rst b/docs/source/version.rst index 02e4e60..4bb096f 100644 --- a/docs/source/version.rst +++ b/docs/source/version.rst @@ -14,8 +14,9 @@ Version history Developed at the ConCat lab at TU Berlin. -The code has been restructured and the interprocess communication is now using :mod:`zmq` -instead of :mod:`sqlite`. The dependency on :mod:`yadg` has also been removed. +The code has been restructured and the interprocess communication is now using :mod:`zmq` instead of :mod:`sqlite`. The dependency on :mod:`yadg` has also been removed. + +The driver library is now separate from **tomato**. A :class:`ModelInterface` class is provided to facilitate new driver development. .. codeauthor:: Peter Kraus @@ -32,21 +33,16 @@ instead of :mod:`sqlite`. The dependency on :mod:`yadg` has also been removed. .. sectionauthor:: Peter Kraus -Developed in the Materials for Energy Conversion lab at Empa, in Dübendorf, with -contributions from the THEOS lab at EPFL, in Lausanne. +Developed in the Materials for Energy Conversion lab at Empa, in Dübendorf, with contributions from the THEOS lab at EPFL, in Lausanne. -First public release, corresponding to the code developed for the BIG-MAP Stakeholder -Initiative Aurora, Deliverable D2. Includes: +First public release, corresponding to the code developed for the BIG-MAP Stakeholder Initiative Aurora, Deliverable D2. Includes: - driver for BioLogic devices; - a dummy driver for testing; - basic scheduling/queueing functionality; - data snapshotting and parsing. -This project has received funding from the European Union’s Horizon 2020 research -and innovation programme under grant agreement No 957189. The project is part of -BATTERY 2030+, the large-scale European research initiative for inventing the -sustainable batteries of the future. +This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future. .. codeauthor:: Peter Kraus,