Patch to make ocamlfind relocatable (via a binary installer on macOS, Windows, Snap)

[MSoegtropIMC]

I maintain the Coq Platform project, which provides binary installers for MacOS, Windows and Linux (snap) for INRIA Coq (a proof assistant). In version 8.16 (autumn 2022) Coq switched to use ocamlfind. Now ocamlfind was not designed with relocation by an installer in mind.

I patched ocamlfind such that it works with relocation.

The main changes are:

• ocamlfind searches the directory structure upwards from the executable until it finds a file findlib.root or reaches /.
• The OCAML_SITELIB, OCAMLFIND_CONF are augmented with a _RUN variant which allows to give e.g. relative path to the path of the findlib.root file (three leading dots are replaced with the path of findlib.root).

This mechanism has the advantage, that one need not patch files with directories during installation - one just has to put the findlib.root at the proper place, which is no issue on the supported platforms.

This system should have a few 1000 installations meanwhile on macOS, Windows and Snap and I did not hear anything negative. If this is desirable for other ocamlfind users is a different question.

Please let me know what you think and if you would like to see a PR or if you have a better idea I can also keep this as a Coq Platform local patch - it shouldn't kill me to port the patch.

[gerdstolpmann]

Hello @MSoegtropIMC , first of all, welcome to the Findlib community.

Relocatibility is a feature that is requested from time to time, but I am still very skeptical. I know that it works for a lot of app programs, but I also know situations where it breaks - in particular for incompatibile libraries and when security problems are not tolerable (like for most professional software).

My other remark is that you can already have your own config file and relocate everything there. It is not as convenient because you need an install script, but it is not a big problem either.

[MSoegtropIMC]

Can you point me to the documentation for the runtime config file? This might indeed be an option - I was not aware of such a mechanism.

[gerdstolpmann]

Here you go: http://projects.camlcity.org/projects/dl/findlib-1.9.6/doc/ref-html/r865.html

[MSoegtropIMC]

Ah yes - I think I came across it, but discarded the possibility of using it because of:

The directory containing findlib.conf is determined at build time (by running the configure script), the fallback default is /usr/local/etc. You can set a different location by changing the environment variable OCAMLFIND_CONF which must contain the absolute path of findlib.conf.

I can't have a compile time determined or fixed location to look for findlib.conf.

[MSoegtropIMC]

(Using environment variables is not an option either).

[gerdstolpmann]

Well, the patch you propose will break many other installations, so any relocation feature would have to be optional and enabled somehow. It is a bit unclear to me how to do that if you can't even use environment variables.

[MSoegtropIMC]

The mechanism I am using is to introduce two new configure time variables:

OCAML_SITELIB_RUN
OCAMLFIND_CONF_RUN

which point to the respective locations at run time. If either of these these variables starts with three leading dots "...", these 3 dots are replaced with the location of the findlib.root file. If OCAML_SITELIB_RUN or OCAMLFIND_CONF_RUN are not given at configure time, the default to the non RUN variants and findlib behaves normally (it does some additional path normalisation which should be a nop then). If the variables don't start with 3 dots, the paths are fixed (but might be different between compile and run time).

[dra27]

Just a very brief note that my patches for relocation in the compiler approach the absolute path problem for ld.conf (which is related to findlib's search for findlib.conf) by using a configure'd relative path, rather than a search up from the directory.

For ocamlrun, that means that it evaluates ../lib/ocaml from the directory where ocamlrun is. Could a similar trick be deployed for ocamlfind - so instead of, say, /home/dra/.opam/dev-4.14/lib/findlib.conf being a hard-coded path, it instead has ../lib/findlib.conf? The OCaml patches for that are all opt-in as well - so opam can use it, but the default for Unix distributions and so forth is absolute paths.

[MSoegtropIMC]

I tried this, but it does not work for Coq Platform, because the executables end up ind different folders at different levels.

[gerdstolpmann]

Using a different approach.

[MSoegtropIMC]

Using a different approach.

@gerdstolpmann : can you please elaborate a bit on this? Is there a new approach which could work with relocating installers?

[gerdstolpmann]

I've merged #72, which adds an optional configuration -with-relative-paths-at <prefix>, where <prefix> is the location at initial install time. Later at use time (when some relocation might have happened) the prefix is derived from environment variables. You can also use $PREFIX in findlib.conf.

Yes, it depends on environment variables. I don't see any good alternative that also covers dynamic loading, which is an integral feature of ocamlfind (and in this case the executable can be an arbitrary one).

Can you explain why an installer can't set environment variables?

[MSoegtropIMC]

Can you explain why an installer can't set environment variables?

It gets messy when you install several variants of Coq Platform. It does happen that students visit two lectures at the same university which use e.g. a standard and a custom variant of Coq Platform or require a different version (Coq is less backwards compatible than OCaml and not all lecturers update their material at the same time). In such a situation one first has to know this and second ask the students to constantly switch around the environment variable. The method to do this depends on the system, the used shell, the used IDE (VSCode has its own idea what the environment should be). This is highly maintenance intensive.

I still don't understand what the problem with an optional "search root upwards from current executable" approach is. As long as one assumes that all executables are installed in a common root - which should always be the case with installers or opam - this does work. This does not work with Linux system packages where things are scattered all over the place, but for system installer packages, one should simply disable this option.

[MSoegtropIMC]

Btw.: the patched findlib should have > 10.000 installations on MacOS, Windows and snap meanwhile. So far no complaints.

[gerdstolpmann]

Can you explain why an installer can't set environment variables?

It gets messy when you install several variants of Coq Platform. It does happen that students visit two lectures at the same university which use e.g. a standard and a custom variant of Coq Platform or require a different version (Coq is less backwards compatible than OCaml and not all lecturers update their material at the same time). In such a situation one first has to know this and second ask the students to constantly switch around the environment variable. The method to do this depends on the system, the used shell, the used IDE (VSCode has its own idea what the environment should be). This is highly maintenance intensive.

Honestly, this sounds like a very special problem.

I still don't understand what the problem with an optional "search root upwards from current executable" approach is. As long as one assumes that all executables are installed in a common root - which should always be the case with installers or opam - this does work. This does not work with Linux system packages where things are scattered all over the place, but for system installer packages, one should simply disable this option.

Keep in mind that people want to development new executables with OCaml, and that these do not necessarily end up in the bin directory of opam. E.g. in our development team we defined a custom location for gathering up our own executables. I guess this is not so uncommon, because you want these to be separated from any package manager.

Btw, I'm not completely opposed to using executable-based methods, but they can't be standard methods, because they only work when you can make certain assumptions.

[MSoegtropIMC]

Honestly, this sounds like a very special problem.

IMHO if it would be rare that people want several OCaml installations on one system, there would be no support for opam switches. E.g. I have on most systems several python versions as well.

Btw, I'm not completely opposed to using executable-based methods, but they can't be standard methods, because they only work when you can make certain assumptions.

As I said, I would not make this default - and only switch it on in builds which are meant to be used in a relocatable installer. The patch I proposed requires a few environment variables to be set during configure time to enable it.

I see your issue with creating executables in user folder. But this is usually done on the command line and for command line there are reliable ways to set environment variables. Coq Platform e.g. comes with init scripts similar to what eval $(opam env) does. The issue is more that IDEs work without manually messing around with environment variables.

One can also think about having a findlib.root file which can redirect to a different folder. This is still easier to handle than global environment variables, since one can make it project specific.