Working with /opt

[cgwalters]

(Again writing a question proxying another location)

A question came up about /opt again. We have docs here: https://containers.github.io/bootc/filesystem.html#opt

But how to e.g. work with an RPM that installs to /opt?

[cgwalters]

This of course heavily relates to an old issue: coreos/rpm-ostree#233

Some systems like FCOS make /opt a symlink to /var/opt - it's a writable, persistent directory. This means that one can install software there per-machine, and manage it out of band. But management and updates of the app are up to the admin - and trying to install it as e.g. an RPM will run into conflicts around things like the RPM database being readonly as it's in /usr.

For bootc systems that are derivation points like fedora bootc we ship /opt as a regular directory. This has the opposite tradeoff!

By shipping such content in a derived container build it will "just work" in general to install RPMs that install to /opt as part of a container build - but the directory is read-only at runtime. Also, it means one needs to rebuild the container when either the OS or the app changes.

To deal with applications that expect write access to a subdirectory such as /opt/someapp/logs, the recommendation is to redirect those to a writable location, in this case, ln -sr /var/log/someapp /opt/someapp/logs (or change the config for the app if possible). Alternatively, one can use systemd unit directives like TemporaryFileSystem=/opt/someapp/logs.

[cgwalters]

I've said this elsewhere but a core problem with RPMs that install in /opt is they're really trying to do two conflicting things. They're trying to be independent of the OS (separate binary path, often separate vendored C/C++ library dependencies, separate log dir often, etc). Except, by using RPM they are tied to the OS because there's just one big RPM database (in /usr nowadays) - and obviously they're trying to support dnf upgrade (or equivalent) upgrading both the OS and the app which some sysadmins want.

When trying to go to image based updates, we either need to pick one way or the other. Either it's lifecycled exactly with the OS, i.e. built into a derived container (and ideally binaries etc. are read-only at runtime), its logs go to the journal (or wherever regular logs are configured to go, etc.) or it's managed fully separately.

A lot of similar software has resolved the tension by going fully the latter way: The software updates itself (or has a tool to update it), it isn't updated by dnf upgrade etc. You can still do that model with a bootc system by having /opt go back to being a symlink to /var/opt and then install the software on firstboot and let it run/update itself. That said IMO the CrowdStrike outage shows well the downsides of this model - this type of software is highly privileged and having it update outside of a bootc upgrade is often not what one really wants.

[cgwalters]

Also related to this is ostreedev/ostree#3120 where ostree has experimental first-class support for having directories like /opt be an overlayfs - but in that case it's stateful, which has quite a lot of implications.

I was asked about the idea of just having a transient overlayfs on /opt - from my PoV definitely! It's ultimately of course the same approach as what bootc usroverlay does.

It'd probably make sense to add docs for streamlining this more.

This also intersects with ostreedev/ostree#3177 a bit.

[cgwalters]

Q: Can I make /opt a separate filesystem (i.e. with a .mount unit or fstab entry)?

A: Sure, but note if you put any content in there in the container image then it will just be masked at runtime. In this model then you are managing updates to /opt out of band. Another way to do this that may be cleaner is to physically store this data in e.g. /var/lib/someapp like this:

RUN <<EORUN
set -xeuo pipefail
# Ideally also make a tmpfiles.d unit for this
mkdir /var/lib/someapp
rmdir /opt
ln -sr /var/lib/someapp /opt

If it's problematic for /opt to be a symlink (the real /var/lib/someapp will be visible in e.g. logs for anything that canonicalizes paths) you can make it a bind mount instead with a .mount unit.