nextcloud*: build from source

[onny]

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[onny]

Currently build stuck at npm build phase, don't know why :(

[SuperSandro2000]

that's likely #353709

[provokateurin]

I think this is quite interesting, but I don't see what the benefit is over using the pre-built archives. @onny maybe you can explain a bit what your motivation is?

[onny]

I think this is quite interesting, but I don't see what the benefit is over using the pre-built archives. @onny maybe you can explain a bit what your motivation is?

building from source has the benefits that you can easily patch things or include important security fixes or try feature branches.
on the other hand, building the source in a web app might be the same as building any other software from source. you dont have to trust the tarball that anything in there is correct

[onny]

i also did this for invoiceplane and it is far more transparent and reproduceable with more checksums and stuff https://github.com/NixOS/nixpkgs/pull/353783/files

[provokateurin]

You need to be aware that the release archive is not the same as just the sources plus dependencies plus npm build. It should be doable to reproduce it mostly, but I fear they might drift apart unnoticed in the future leading to problems down the road.

[Ma27]

My personal stance on this is that the xz incident has taught us basically that building from sources is always desirable.

It should be doable to reproduce it mostly, but I fear they might drift apart unnoticed in the future leading to problems down the road.

Can you elaborate on what you mean with mostly?
Not sure, but my gut feeling is that if this drifts apart, that's potentially a bug, no?

[provokateurin]

Can you elaborate on what you mean with mostly?

The release process calls https://github.com/nextcloud/server/blob/master/core/Command/Integrity/SignCore.php and since those are private keys we will never be able to replicate the signature as in the release archive.

Not sure, but my gut feeling is that if this drifts apart, that's potentially a bug, no?

Yes, but it could result in quite fatal ones. For example if a new app is added an shipped, but we don't notice it and something important relies on it, then we might break instances completely. I agree it's "just" a bug, but nevertheless something to consider.

The release script is not public and I'm not sure if it is possible to fully understand everything it does without seeing it yourself. Maybe comparing the archive and the derivation is easy enough to eventually figure out everything that has to be done in the derivation, but that will be annoying work.

[provokateurin]

Using diffoscope might be enough to reproduce the entire build process without having to rely on guessing or private information.

I might give this a shot some day, maybe upstream could make use of it in some capacity as well ;)

[Ma27]

Yes, but it could result in quite fatal ones. For example if a new app is added an shipped, but we don't notice it and something important relies on it, then we might break instances completely. I agree it's "just" a bug, but nevertheless something to consider.

That's fair.

I wouldn't want to diverge from what upstream recommends, to me the risk of subtle runtime-only issues is too high in this case.

My point was mainly that this is a desirable property and I wanted to understand the reasoning against it, so thanks for elaborating.

Using diffoscope might be enough to reproduce the entire build process without having to rely on guessing or private information.

I might give this a shot some day, maybe upstream could make use of it in some capacity as well ;)

I'd expect that to help with making a good call here, so this would be highly appreciated. Thanks!

[provokateurin]

So should we eventually also build all apps from the store from source? This would mean a lot of maintenance work and would slow down updates a lot.

[provokateurin]

For me npm ci is stuck as well and it's not the kernel bug since I run 6.14.6.
I also tried to check with strace what is going on, but it just stops at a read syscall and then nothing happens anymore.

[provokateurin]

I just started looking into this again and was able to make some great progress.
It's quite complicated also building all the default shipped apps from source.

[provokateurin]

Replaced by #442910. Thanks for your initial work @onny!