Skip to content

Roadmap

Jump to bottom
Sebastián Mancilla edited this page Mar 14, 2019 · 15 revisions

Plugins should provide info file about their services

The services.yaml file currently requires the user to know the actual classpath/library of the service in order to use it as part of the application.

# services.yaml
io-services:
  reader:
    class: org.jlab.clara.demo.services.ImageReaderService
    name: ImageReaderService
  writer:
    class: org.jlab.clara.demo.services.ImageWriterService
    name: ImageWriterService

services:
  - class: org.jlab.clara.demo.services.FaceDetectorService
    name: FaceDetectorService
  - class: pupil_detector_service
    name: PupilDetectorService
    lang: cpp

The user should not need to be aware of the implementation details of the services nor their actual location on disk.

At the root of the plugin directory there should be an info.yaml file (TODO: decide the right name), which lists all services provided by the plugin:

# info.yaml
plugin:
  services:
    - class: org.jlab.clara.demo.services.ImageReaderService
      name: ImageReaderService
      lang: java
    - class: org.jlab.clara.demo.services.ImageWriterService
      name: ImageWriterService
      lang: java
    - class: org.jlab.clara.demo.services.FaceDetectorService
      name: FaceDetectorService
      lang: java
    - class: pupil_detector_service
      name: PupilDetectorService
      lang: cpp

This frees the user to know the details about the services and allow them to use just their registration names to compose an application:

# services.yaml
io-services:
  reader:
    name: ImageReaderService
  writer:
    name: ImageWriterService

services:
  - name: FaceDetectorService

  - name: PupilDetectorService
    lang: cpp

CLARA can then map each service name to the proper class to be loaded by using the information provided by the plugin itself.

TODO: Decide if lang should be a required key or not. The language information should be provided by the info.yaml file, but it is also useful that the language is stated clearly in the services.yaml.

TODO: Services installed individually (i.e. not part of a plugin) should also provide some kind of info.yaml file, but currently there is not specific layout for single services.

TODO: info.yaml should list plugin-provided orchestrators too?

Finish clara-shell API to add custom Java-based commands

The main idea of implementing the shell in Java was to support extensibility. Plugins should be able to add custom Java-base commands and custom variables. Most of the pieces are already present in the code and the feature just needs to be finished.

The plugin should provide a factory class that will register commands and variables within the shell. With support for info.yaml in the root of the plugin, the factory can be defined as:

# info.yaml
plugin:
  shell:
    factories:
      - class: org.jlab.clara.demo.shell.DemoFactory

Multiple factories should be supported. On startup, the shell should load all available factories.

This feature would enable running CLAS12 reconstruction and analysis directly from the shell, if analysis commands are provided.

TODO: Make loading optional? Register the command but actually load the class the first time the command is used?

TODO: Register commands and variables directly from the YAML.

TODO: Test launching GUIs apps.

(Optionally) Support running scripts/binaries from the shell

It should be possible to run binaries directly from the shell. Proposed syntax:

clara> !custom-script

clara> !some-binary --args

TODO: Add each <plugin>/bin directory to the PATH inside the shell?

(Optionally) Allow modifying the environment inside the shell

Provide a new command to set environment variables from the shell:

clara> setenv MY_VAR 1

This environment variable should be available to all commands that run from the shell.

(Optionally) Mark shell variables as advanced

There should be a flag to mark a variable as advanced. Then the variable would not appear when the show command is used normally. This would help reducing noise when too many variables are defined.

Orchestrator

WIP.

Use isolated classloaders to load plugin/services

Currently the entire class loading is handled by using the same classpath for CLARA and all plugins/services, which is roughly like this:

<clara>:<plugin1>:<pluginB>:...:<services>

This creates all sort of problems when multiple plugins are involved and different versions of dependencies are required.

A new classloader must be created each time a service is deployed. This classloader should only use paths related to the service/plugin:

  1. $CLARA_HOME/plugins/<plugin>/services/<service>/classes
  2. $CLARA_HOME/plugins/<plugin>/services/<service>/lib
  3. $CLARA_HOME/plugins/<plugin>/services (legacy directory)
  4. $CLARA_HOME/plugins/<plugin>/lib
  5. Plugin defined custom directories inside $CLARA_HOME/plugins/plugin.
  6. $CLARA_HOME/lib (for Engine and EngineData) (*)

In the case of individual services (installed into $CLARA_HOME/services directory) it could be:

  1. $CLARA_HOME/services/<service>/classes
  2. $CLARA_HOME/services/<service>/lib
  3. $CLARA_HOME/lib (for Engine and EngineData) (*)

Since CLAS loves to have their own directory layout (which requires us to support those paths directly when launching the DPE) there should be a configuration key inside info.yaml that list the directories inside the plugin that must be added to the classpath (and clean j_dpe from CLAS specific paths):

# info.yaml
plugin:
  classpath:
    - core
    - clas

But in general, a common directory layout should be recommended.

TODO (*): CLARA jars and dependencies are all together inside $CLARA_HOME/lib. Maybe separate them between public and private?

Clone this wiki locally