How do you use the sandbox store?

[chrisburr]

From the BiLDx meeting.

TODO: Add more description.

[marianne013]

We currently have a dedicated machine for the sandbox store that's part of our dirac setup:
dirac01: Everything but the sandbox store
dirac02: sandboxes and backup CS
diracweb: web interface/pilot.json etc
diracdb: all the data bases
We don't use a grid storage elements for sandboxes at all. Our sandbox size is set to 20 MB max.
We've seen very occasional overloads, usually where users weren't aware that they could upload their results to a SE and download it from there.

[chrisburr]

LHCb runs the sandbox store with the LocalSE on a single machine. Jobs are cleaned after ~1 week so we only care about sandboxes until the jobs are deleted.

[marianne013]

We do the same (delete the sandboxes when the jobs go).

[andresailer]

Same for iLCDirac

[iueda]

Belle II:
Currently we run SandboxStore on a single DIRAC server node, with the files on the local disk (SSD).

However, we often suffer from the SandboxStore overloaded (limit from disk i/o) and have to manipulate
the JobScheduling parameters (MatchingDelay, RunningLimit), so that sandbox accesses would not fail with timeout.

Due to this limit, when we do not have long-running production jobs, and short jobs are dominant, we cannot
fill the computing resources, and there are complaints.

We are trying to utilize an SE, or a set of SEs as SandboxStore, with specifying sandbox files with 'lfn:'
We may keep the SandboxStore on the DIRAC server node as the principal one, and push only some specific sandox files to SEs.
We have different workflow around sandbox for production jobs (made by transformation) and for user jobs (submitted by the individual users)

We have observed output sandbox files uploaded onto SEs (as fail-over), that might have helped those jobs without us noticing.

[chrisburr]

I have a few questions:

• How big are your sandboxes typically? (I guess this should be split by input vs output and user vs production)
• How many jobs do you have per day?
• Do you know anything about the specs of the SSD? Is it a relatively new NVMe drive or something older?

[hmiyake]

Sorry we haven't answered your questions.

We don't have exact statistics but here are typical numbers:

As for sandbox size,

• Production job: input (~10KB + ~2KB), output (~10KB)
• User job: input input (10KB ~2M), output (5 ~10KB)

Note that we have two input sandbox for production job, one is common among a Transformation, the other is generated at each job submission.

As for number of jobs, I'd like to show job execution rate

• Production job: max. 5K/h, ave. 1.5K/h (so roughly ~36K/d)
• User job: max. 17K/h, ave. 7.1K/h (~170K/d)

Regarding as sandbox local strage, turned out we don't use SSD but HDD RAID box.
Performance measured from the monitor is ~1000 IOPS.

Honestly we didn't recognize our RAID box spec...at least we can expect some amount of improvement by transition to SSD (yes hopefully NVMe...)

[fstagni]

Hi @hmiyake , do you have an update on the performances of your sandbox disk?

The reason why we are asking this question is for knowing if anyone really need the "external SE" functionality of the Sandbox in DIRAC today. For reasons of migration to DiracX (where the sandboxes will be stored in an object storage) we would clearly prefer to not have to take the case of external SE into consideration.

[iueda]

Hideki should respond to the above "performances" question, but let me understand the aim for the "drop" of the "external SE" functionality of the Sandbox

As I can see two different aspects in the use cases of sandbox files on external SE:

1. The support for remote SE to be used as the sandboxstore, such as this;
   https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/WorkloadManagementSystem/Service/SandboxStoreHandler.py#L166-L172

2. Jobs on WNs downloading "sandbox" files specified with LFN from SEs
   https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/WorkloadManagementSystem/JobWrapper/JobWrapper.py#L1155-L1171

Our initial idea of trying an SE, or a group of SEs, was to use it/them as the standard sandboxstore (1), but
currently we are trying

• to put "standard" sandbox files on the standard sandboxstore (local DIRAC service), and
• to put additional sandbox files on SEs, and let jobs download them as supplement to the standard sandbox files

If this workflow gives us sustainable load on our sandboxstore, what we would need is not (1), but only (2) and the related functionalities.

Is it just (1) that you would like to drop, or everything including (2)?

[fstagni]

We would just like to drop (1)

[chrisburr]

To add, LHCb makes heavy use of (2) for user jobs.

[fstagni]

Hello, I simply "ping" on this discussion as knowing if simplifications are possible would speed up the process. @hmiyake more details?

[hmiyake]

Terribly sorry for long silence...

We will have our computing system update in this year (2024) but unfortunately we cannot expect significant performance improvement (e.g. replacement with SSD RAID box)

Once we considered (1) feasibility but now we have neither activity nor plan for that.
On the other hand, we are implementing (2) for widely usage in user analysis.

We then agree to drop (1) in DiracX.