-
Notifications
You must be signed in to change notification settings - Fork 0
GridPP DIRAC: A quick introduction
Note: This information was previously hosted on the GridPP Wiki.
If you are completely new to grid computing, please either talk to computing support in your experiment and/or your local grid admin first to make sure you have the basics covered. Any attempts at producing a "one size fits all" documentation have failed, for good reason. If you have no idea who to contact, please email tb-support/at/jiscmail.ac.uk with as much detail as you are able to provide and we'll work it out from there.
These instructions refer to the GridPP DIRAC server at https://dirac.gridpp.ac.uk/. Other DIRAC servers might be configured slightly differently. (Yes, this sounds a bit like the spiel you get at the start of a flight.)
This document describes how to quickly set up a DIRAC UI client, create a job script and then submit it to the GridPP DIRAC server for execution. You need to have a valid grid certificate* and be a member of a GridPP supported VO. You do not need root access to the machine you are installing the DIRAC UI on. If your machine has access to cvmfs, there's also a cvmfs based UI you can use instead. Please see below for details.
*By popular request: If you have your grid certificate in a p12 format, you need to convert it to pem files, and place those in the .globus directory in your home directory.
openssl pkcs12 -nocerts -in mycert.p12 -out userkey.pem openssl pkcs12 -clcerts -nokeys -in mycert.p12 -out usercert.pem chmod 400 userkey.pem (this is a security requirement)
- Please consider signing up to the gridpp dirac users mailing list: Sign up! (It's fairly low traffic.) If you are using a non-institutional email, please let us know, it's hard to separate the spam from the real requests. Thank you.
- If something isn't working, please try the mailing list first. It may help to include the output from your commands using the debug flag, which can be done with the
-dddflag, e.g.: dirac-proxy-init -ddd -g gridpp_user -M - For operational issues please consider filing a GGUS ticket. If you set "Notify site" to UKI-LT2-IC-HEP, we'll see it quicker. If you don't know if your issue is operational, just use the mailing list :-)
- If your jobs don't run at a specific site that you think they should be running at (but run on other sites), please email lcg-site-admin/at/imperial.ac.uk. Please include your DN and VO. We will then check if it is a configuration issue on the DIRAC side and either fix it, or advise you whom to contact instead. Alternatively you can use GGUS for this issue. Please set "Notify site" to UKI-LT2-IC-HEP, we will reroute it to the appropriate site, if applicable.
To be able to use DIRAC you need to be registered with a VO and the DIRAC server. Once you are registered with a VO, this will be automatically propagated to the DIRAC server, but there can be a couple of hours before all systems have synchronised. To check that your registration is complete, please ensure that your certificate is correctly imported into your browser and go to the GridPP DIRAC server's web interface. In the bottom left corner, click the little '+' next to settings and check your group. If this shows you as "visitor" and you cannot change this, you are not registered. In this case, please contact lcg-site-admin/at/imperial.ac.uk
If you do not want to install a client and have cvmfs available on your machine you can do:
source /cvmfs/dirac.egi.eu/dirac/bashrc_gridpp dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
and skip the rest of this paragraph.
Note: The installation runs in a conda environment and is self-contained, it should work on most Linux platforms. We recommend Rocky or Alma 9.
Your user cert and key (usercert.pem/userkey.pem) should be located in the $HOME/.globus directory. You also need to be a member of a GridPP supported VO to be able to install a UI.
Please check that your firewall is open on the appropriate ports:
nc -v dirac01.grid.hep.ph.ic.ac.uk 9135
Connection to dirac01.grid.hep.ph.ic.ac.uk 9135 port [tcp/*] succeeded!
Installing a DIRAC UI
mkdir dirac_ui cd dirac_ui curl -LO https://github.com/DIRACGrid/DIRACOS2/releases/download/2.42/DIRACOS-Linux-x86_64.sh bash DIRACOS-Linux-$(uname -m).sh rm DIRACOS-Linux-$(uname -m).sh source diracos/diracosrc pip install DIRAC==8.0.48 dirac-proxy-init -x -N (ignore the message about "No CRL files found" - these will be installed with the next step) dirac-configure -F -S GridPP -C dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server -I dirac-proxy-init -g [your_vo_goes_here]_user -M # (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
If you see an error like the one below then you probably are not registered with the DIRAC instance.
Could not sync dir Cannot get URL for Framework/BundleDelivery in setup MyDIRAC-Production: Option /DIRAC/Setups/MyDIRAC-Production/Framework is not defined
- Create a proxy
If you don't already have a valid proxy you should create one:
source diracos/diracosrc (or source /cvmfs/dirac.egi.eu/dirac/bashrc_gridpp if using the cvmfs ui) dirac-proxy-init -g [your_vo_goes_here]_user (e.g. dirac-proxy-init -g comet.j-parc.jp_user -M)
- Create a JDL
Now we are ready to create a simple DIRAC jdl. Put the lines below into a file Simple.jdl:
[
JobName = "Simple_Job";
Executable = "/bin/ls";
Arguments = "-ltr";
StdOutput = "StdOut";
StdError = "StdErr";
OutputSandbox = {"StdOut","StdErr"};
]
- Submit a job
$ dirac-wms-job-submit -f logfile Simple.jdl JobID = 236
- Check the job status
$ dirac-wms-job-status -f logfile JobID=236 Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY
- and eventually:
$ dirac-wms-job-status -f logfile JobID=236 Status=Done; MinorStatus=Execution Complete; Site=LCG.UKI-LT2-IC-HEP.uk
The jobs can also be monitored using the Web interface. Go to Jobs ->Job monitor
- Once the job is done, retrieve the output:
$ dirac-wms-job-get-output -f logfile
A list of JDL parameters used by DIRAC can be found in the DIRAC documentation.
Jobs running within DIRAC will start with an environment similar to the python3 DIRAC UI; this means that by default:
- All dirac-* commands (such as dirac-dms-add-file) are available within job scripts.
- Running "python" will start python3 with the DIRAC API ("import DIRAC") module available.
- Other grid tools, such as gfal_utils (gfal-copy, etc.) are available in a configured and working state.
- A few other commands (curl, openssl, perl) will use the DIRAC versions by default. You may need to call the system versions with the full path if you specifically need the system version for any reason.
It is recommended that you run one of your jobs locally while you have the DIRAC UI sourced and check that everything works before submitting to the grid. As always, if you find any problems you need help with, please contact the gridpp-dirac-users list.
For any real work, we recommend that you develop your code against the DIRAC python API (see below). Please also read the data handling within a job page before you produce any kind of data to be stored on a storage element (SE). We generally encourage to use DIRAC's inbuilt data management tools.
An example of a hello world job can be found
here.
Another example with more options: Options!. You will also need this shell script which is used as an executable within the example.
The corresponding section in the DIRAC user guide can be found here and the Job API here.
Currently most sites are providing either Rocky 9 or Alma 9 on their WNs, commonly referred to as 'EL9' ('Enterprise Linux 9') in DIRAC. A few sites are still providing either EL7 or EL8 based WNs, to select those, you can use:
Platform = "EL7";
If your jobs are containerized and will run anywhere you can specify AnyPlatform as the platform name to disable this check:
Platform = "AnyPlatform";
Note that all parts of this JDL string are case sensitive (the word Platform and the platform string itself).
Users of the DIRAC API can use the "job.setPlatform("EL8")" function to configure this behaviour.
Ganga users can set extra options, including the platform, via the backend settings attribute:
j = Job(backend=Dirac()) j.backend.settings['Platform'] = "EL9"
If you want to run at a specific subset of available sites for your VO, you can specify a list of Sites using the usual Site clause with a list:
Site = {"Site1", "Site2", "Site3"};
It is also possible to select specific CEs (generally this is used for targeting special resources):
GridCE = {"ceprod05.grid.hep.ph.ic.ac.uk", "ceprod06.grid.hep.ph.ic.ac.uk"};
Note that selecting an invalid combination of Sites & CEs may result in your job staying in the waiting state forever.
Note: If you are ready to run some real workloads, please see the data handling within a job section on how to best handle your outputs.
List files in the DIRAC file catalogue (please do not use the file catalogue for anything else):
$ dirac-dms-filecatalog-cli Starting FileCatalog client File Catalog Client $Revision: 1.17 $Date: FC:/> ls (use 'exit' to, well, exit)
Add a file to an SE (here UKI-LT2-IC-HEP-disk). Please replace /gridpp with the name of the VO you are using, and firstname.lastname with your first and last name (or alias or superhero, it just has to be unique):
dirac-dms-add-file /gridpp/user/f/firstname.lastname/myfile.txt myfile.txt UKI-LT2-IC-HEP-disk
List all replicas:
dirac-dms-lfn-replicas /gridpp/user/f/firstname.lastname/myfile.txt
Copy file from an SE to disk:
dirac-dms-get-file /gridpp/user/f/firstname.lastname/myfile.txt
Replicate file from on SE to another (file is currently at UKI-LT2-IC-HEP-disk):
dirac-dms-replicate-lfn /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-QMUL2-disk
Ask for the access URL (you should only need this for very special cases):
dirac-dms-lfn-accessURL /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk
Remove file from a specific SE:
dirac-dms-remove-replicas /gridpp/user/f/firstname.lastname/myfile.txt UKI-LT2-IC-HEP-disk
Remove all replicas (?):
dirac-dms-remove-files /gridpp/user/f/firstname.lastname/myfile.txt
Using the asynchronous files transfer tools.
Replicate a file to a given SE:
dirac-dms-replicate-and-register-request [make up an id for this transfer] [LFN] [target SE]
e.g. replicate a file to the RHUL SE (files must exist and be registered in the file catalogue):
dirac-dms-replicate-and-register-request daniela1 /gridpp/user/d/daniela.bauer/small.text.file.txt UKI-LT2-RHUL-disk
Querying the transfers:
dirac-rms-request [transfer if from above, e.g. daniela1] dirac-rms-request --Status='Done' --Since='2016-05-05'
Resetting a transfer (this will only resubmit the failed transfers:
dirac-rms-request --Reset [transferid]
The DIRAC File Catalog allows you to associate metadata with your files. You can find a basic introduction here:
DIRAC File Catalogue Metadata
Note that currently metadata is not heavily used, please report any bugs to us, so we can try to fix them.
Please see the documentation here: Data handling
Experimental GPU Support
For details of how to use GPUs with DIRAC see Note on GPU support
DIRAC command reference
DIRAC command reference on ReadTheDocs
Back to Home.