Calling another build system from dune with a job limit

[rlepigre]

I have a situation where I need to call make to build a large set of files, so currently I'm doing something like the following (simplified):

(rule
 (targets %{read-lines:targets.txt})
 (deps ...)
 (action (bash "make -j4 ...")))

The problem is that if I do not specify a -j option for make, then the build takes ages. On the other hand, one might want to tweak the given number of jobs to the number of available CPUs. Another issue is that if dune runs with N processes, then at some point during the build there may be up to N - 1 + 4 jobs, which is not very nice. In my case, I actually have several such rules, so the situation is even worst.

I thought I might be able to use the DUNEJOBS variable here, but I don't think that's true.

If there is no solution, it could be nice to extend actions DSL with a (with-jobs N ...) stanza, that specifies that N jobs need to be allocated for the action. Any thoughts on that?

[ejgallego]

Hi @rlepigre , this has been discussed quite a few times (don't have time now to dig old discussions issues, but I think it was titled "jobserver protocol")

As far as I know, there is not easy solution yet, but indeed, it would be great to be able to implement something in the lines you propose.

[SkySkimmer]

#4331

[tbrk]

From what I could understand, the jobserver protocol feature was not integrated. Is that right?

What would be wrong with the idea of requesting some parallelism in a rule, e.g., (max_parallel_jobs 12), and then having a variable return the granted parallelism, e.g., (run cmake … -j %{granted_parallel_jobs})? The idea being that dune would decide, based on the constraints communicated by opam and the dependency graph, how much parallelism to dedicate to a vendor build using, for example, cmake.

Obviously, someone still needs to implement all this, but would the idea be useful and acceptable?

[Alizter]

I should add that this isn't to say that such a feature is impossible to support in dune, it's in fact probably quite easy to support. The main difficulty is with the idea itself and the footguns it introduces. This kind of problem is hard, and every proposal has its own downsides. To make the case for dune, we would need a proposal that is difficult to abuse, easy to use and yet useful enough to justify the effort to implement it.

[tbrk]

My naive idea was for a rule to declare its parallel “potential” (max_parallel_jobs) even though it may have to run with less. Then when dune decides to execute the rule it decides how much of its available parallelism to dedicate (taking into account other rules that could run in parallel) and communicates that value to the rule (max_parallel_jobs). I agree that some kind of heuristic would be needed to make this choice, there being two simple extremes: max_parallel_jobs = 1, which is de facto the case today for “well behaved” rules, and max_parallel_jobs = $OPAMJOBS, if dune decides to run an external build to completion before continuing with other rules. The latter would already be better than the current state of affairs, IMHO.

[rgrinberg]

Suppose you have the following action:

(progn
 (run make -C foo -j %{granted_parallel_jobs})
 (run make -C bar -j %{granted_parallel_jobs}))

Would you expect the variable to evaluate to the same value?

[tbrk]

For me, it would be the same value for both since the “job allotment” would be by rule.

[rgrinberg]

That makes sense semantically, but I think it makes the feature less useful. It seems like an action should be able to get a fresh quota every time it asks for one.

[rgrinberg]

If there is no solution, it could be nice to extend actions DSL with a (with-jobs N ...) stanza, that specifies that N jobs need to be allocated for the action. Any thoughts on that?

Shouldn't this annotation go on the underlying actions that consume these jobs? So run, bash, system etc. (with-jobs 5 (echo ..)) wouldn't make sense.

[tbrk]

A dedicated action seems more elegant and flexible than a per-rule option. It could have the same request/grant semantics (jobs-request 5 (run make -C foo -j %{jobs-granted}), with 1 <= %{jobs-granted} <= 5. Or with a binding: (jobs-request 5 (lambda (granted) (run make -C foo -j granted)); too complicated? I somehow missed @rlepigre's second suggestion, which would be more (jobs-required ...).

[tbrk]

Even though it doesn't make much sense to reserve jobs for an (echo ...), the orthogonality could be worth it. Dune could raise an error or warning if the scope of the request does not contain run, bash, system, etc., or simply grant only 1 job.

[rgrinberg]

To summarize what I don't like about each proposal:

1. with-jobs n allows for deadlocking with -j1 unless it is modified in the way that you've mentioned with a lambda
2. %{jobs_granted} as a rule variable doesn't allow an action to get a fresh quota

Or with a binding: (jobs-request 5 (lambda (granted) (run make -C foo -j granted))

This proposal is what looks the most attractive to me.

Even though it doesn't make much sense to reserve jobs for an (echo ...), the orthogonality could be worth it. Dune could raise an error or warning if the scope of the request does not contain run, bash, system, etc., or simply grant only 1 job.

Even though it doesn't make much sense to reserve jobs for an (echo ...), the orthogonality could be worth it. Dune could raise an error or warning if the scope of the request does not contain run, bash, system, etc., or simply grant only 1 job.

Granting one job is worse than the current behavior though. Currently, we just run echo without reserving anything. Anyway, if you're prepared to live with the downsides, I think it's alright.

[tbrk]

By “deadlocking with -j1”, are you thinking of a scenario where dune is invoked with only 1 job and that some rule requires a minimum amount of parallelism to run? If so, would it be useful to allow rules to request both a minimum and maximum number of jobs, i.e., (with-jobs [2, 5] ...), with an explicit failure if the lower bound cannot be met?

For me, running any action requires at least 1 unit of parallelism, i.e., “the current thread”, but, I have no idea about dune's internals.

[rgrinberg]

If so, would it be useful to allow rules to request both a minimum and maximum number of jobs, i.e., (with-jobs [2, 5] ...), with an explicit failure if the lower bound cannot be met?

That's an option. Although I would prefer to simplify not allow rules that require -j > 1 just to terminate.