Dev Meeting 2020.12.04 (esy drom)

Present: @avsm, @lefessan, @claret, @timbertson, @samoht, @kit-ty-kate, @rjbou, @MisterDA, @ManasJayanth, @AltGr, @dra27, @emillon

@avsm presented some slides on the current status of opam LINK THEM

Key principle at the moment is that OCaml packages need an opam file, but we don't mandate that the opam client is used. With opam 2.2, looking at first changes to the file format since 2.0.

Three major clients so far, apart from opam: Esy (1 in 5 OCaml users from 2020 survey noted that they use esy to access opam-repository), opam2nix (previously demonstrated) and, just announced, Drom

@ManasJayanth introduced Esy: presently maintaining (originally written and maintained by @andreypopp). Key difference between opam is that while packages are consumed from opam-repository exactly as with opam, when creating/developing a package with Esy, the integration is coupled with JavaScript. So esy inherits npm's expectation of local dependencies. Esy borrows from nix the use of hashes to make the installation area for artefacts immutable. This requirement adds to what's needed from opam. Esy also has to provide first class language support for OCaml, Reason and C. Windows support is much more important: many JavaScript users expect Esy to work from PowerShell/CMD, not Cygwin shell. It's solved - as in fdopen's Windows fork of opam-repository - by wrapping the Cygwin installation and largely hiding it from the user. Esy wraps these overlays.

@avsm - what's keeping Rust from first class language support? Is it that Cargo is both build tool and package manager? @ManasJayanth - the problem at the moment is more the linking step.

Present developments are trying to head towards a single source of truth for both packaging and build - in Esy's case, package.json. A more recent project pesy (https://github.com/esy/pesy) started out as a JSON wrapper for Dune, but now seeks to unify the packaging and build under the same tool. @avsm - even in Rust/Cargo community, there's a starting realisation that packaging and build in one pass necessarily works (that there is a distinct stage for gathering packages vs building them).

@ManasJayanth: what's done in Esy (heavily borrwed from Nix): packages must be submitted in Esy format (with opam as the only exception), so for other languages the packages must be repackaged for Esy. This solves the ordering problem: a Python package, for example, requires an additional package.json/esy.json which then allows it to be installed in Esy's sandbox. Main aim with pesy at the moment is to try to push the common rules for the build back into patterns in package.json. @avsm - can a similar process as opam2nix does be used for multi-language support? @timbertson - yes, indeed it looks like Esy packages should be translatable back to Nix as well. @ManasJayanth: it's been necessary, especially wrapping other languages, to impose build hygiene (for example, older OCaml packages which build in-tree). @samoht - does Esy depend on the opam client libraries at the moment, or have to re-implement things? @ManasJayanth - Esy heavily depends on the published opam libraries, yes. @samoht - for fixing packages building out of tree, and hygiene and so forth, is it possible to be pushing these changes back upstream? @ManasJayanth: yes, it's certainly an ecoysystem issue, rather than a tooling issue. So rather than expecting other ecosystems to move, it's solved with overlays. @samoht - indeed, overlays solve, but it's also interesting to see what other tools might want to be fixed in order to make this happen.

@ManasJayanth: for example, dune subst and opam's substs fields. So there are various packages which are published with the version as a tag which is then substituted later. In the VSCode plugin, for example, Dune various things around in order to allow npm to find packages more easily.

@lefessan introduced Drom. It has two purposes: the first is as a scaffolding tool when putting new projects together. For example, you have to put the dune-project together, possibly an opam file, there are CI scripts. The idea was to have a tool which would set this up for new projects. Inspired by Cargo, the idea was to have a similar tool for OCaml. Its second purpose it to operate as a "simple" layer over the top of dune and opam. So if you are not using expert commands, the principle is that drom build and drom install will correctly drive both dune and opam to set-up a local switch, pinning and pulling in dependencies.

@avsm - we took the duniverse tool (which sprang out - and distracted from - an effort on a Cargo-like tool a few years ago!). That's just been renamed to opam-monorepo which allows it to be driven by opam as a plugin. Have you considered doing this for drom, so to become opam-drom?

@lefessan - considered this, yes, but there was the issue that this increases the idea that the user needs to know more about opam. A confusion arises as a plugin because the plugin ends up tied to the switch it was compiled in.

@avsm - this leads to a question for opam 2.2 of whether opam could be distributed already initialised, reducing the dependency on global state. Target for the VSCode plugin is that everything will be installed in the plugin without any requirement for host changes.

@dra27 - for opam 2.2, ongoing work for both layered switches (to allow bin directories) which will address some of this. Also for plugins, looking at the possibility of having a separate list of remotes for maintaining the plugins.

@avsm - could we be looking eventually at removing global switches completely - so eliminating the opam init step entirely.

@dra27 - essentially this would wrap opam init inside opam switch create with the entire OPAMROOT in _opam?

@avsm - yes, but even just wrapping the commands simplifies the entry point for users. @timbertson - and even with local switches at the moment, it's often hard to be sure that the local switch isn't changing global state. @emillon - and the shell state getting out of sync is also a common gotcha for beginners.

@AltGr - indeed, at the moment the issue is that repositories remain global. The repository CLI also makes handling this tricky at the moment.

@avsm - the problem in opam is that existing users are used to the global switches and shell state. Going all-in on local might alienate existing users? @dra27 - not necessarily, at the moment we have the situation where global opam is the default, but local switches are there. We could design around the default being local switches, and have the global state as the next step.

@lefessan - one thought discussed already if we end up with a 3.0 file format might be switching the actual format to one which is easier to interoperate with with other communities.

@avsm - only thing is to ensure that we can move seamlessly between the two.

@samoht - moving to a new file format is good, but should be done with care to ensure that the design decisions behind the original format (infix operators for formulae, and the idea of optional parts - for example being naturally able to specify a package without a formula).

@avsm - are there any other issues with the internals of opam which cause pain? @timbertson - what would be useful at the moment would be to be able to separate out the build instructions i.e. serialising the action plan of what will be done. @dra27 - yes, this is locking the action graph rather than the package solution.

@avsm - what are the current plans for the solver next year? @AltGr - possible plans to replace the C++ parts of the stack with an OCaml part. Also want to be improving the conflict messages further over what's already in 2.1.

@avsm - opam-repository has seen a big update in CI in the last year. We now have a build cluster working behind both the base Docker images for OCaml+opam. The cluster is now being used for testing opam-repository PRs, maintaining opam-repository health and, just this week, starting to generate package documentation for the entire development. The cluster is available for other things too!

Follow-ups:

• Separate meeting to complete the Esy discussion!
• Trial integration of drom as an opam plugin and what might need to change
• Potentially trying to hook opam2nix into CI to be able to test whether packages are Nix-ready