add adaptative job concatenation to mode=grid
#1391
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rvhonorato
commented
Sep 18, 2025
rvhonorato
force-pushed
the
1385-implement-adaptative-efficiency-submission-algorithm-for-modegrid
branch
from
e5f10c4 to
1e9a767
Compare
rvhonorato
force-pushed
the
1385-implement-adaptative-efficiency-submission-algorithm-for-modegrid
branch
from
43dafa1 to
c4978de
Compare
…ion-algorithm-for-modegrid
rvhonorato
added
the
execution
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
You are about to submit a new Pull Request. Before continuing make sure you read the contributing guidelines.
Checklist
DOES NOT APPLY
Summary of the Pull Request
This PR overhauls the
libgrid, it adds an abstract classGridInterfacethat holds the "primitive" methods to interact with dirac. It also addsCompositeGridJobwhich is a representation of a batch grid payload.The
GridSchedulerhas been updated with a feature to "probe" how busy the grid is, it will check how long a specific payload spent in waiting and running, use this to calculate what would be the ideal number of executions that should be concatenated into a single payload to increase efficiency.With these changes the
libgridbecomes adaptative, reacting to the load on the grid network and with this addition we should be able to efficiently process a large volume of jobs. This feature is relevant to ofer haddock3 as a service via wenmr.science.uu.nl/haddock3Related Issue
#1385
Additional Info