Schedule jobs to SE (RFC)

RFC #12

Authors: R.Graciani, A.Casajús

Last Modified: 2013-03-13

Motivation

DIRAC job scheduling has mainly been driven by directing jobs to sites. Although this mechanism suites most use cases, there are several in which that is not enough. The main use case is to be able to delay up to job matching time the decision on which Sites are considered near enough to a given Storage, allowing to attach/detach CPU power as needed. Here we present a proposal:

Goal of the proposal

Users can define a list of SEs from which the Input Data (ID) has to be read or retrieved from. An example use case would be a VO that would like to divide the data processing based on the SE performance and not the processing power provided by Sites.

The proposal also takes into account that data can be either read directly from the SE or downloaded into the Worker Node for local processing. Users should be able to select how they want to access the data.

Proposed implementation

Users can define in the job manifest two options:

• AllowDataDownload: Job can always run at sites that have local access to the ID. This option requests also run at a Site where the ID can be downloaded from a SE previous to the execution of the application. By default is false.
• InputSE: Restricts the SEs where the ID is to be accessed from. All the ID have to have a replica in one of the listed SEs. By default this value is undefined.

When the job goes through the optimization chain, if InputSE is defined, only replicas in the provided list of SEs will be considered.

At matching time, the JobAgent presents a list of Local and Close SEs (Close SEs being those from which jobs can download data). If the manifest required InputSE these will be matched against the ones presented by the JobAgent. If AllowDataDownload is set to True all the SEs presented by the JobAgent will be considered, otherwise only the Local ones will be used for the Matching.