blender: Switch to fetching source git instead of source tarball

[the-furry-hubofeverything]

Description of changes

Splitting up #257780, part 1 (I'm probably going to rewrite some of it, now that I know a bit more about nixpkgs)

This switches from building with a source tarball to cloning the repository directly. From what I understand, this is common practice, but it's a lot slower.

This is a much smaller change so hopefully it doesn't get stalled like my last attempt.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[the-furry-hubofeverything]

@ofborg build blender blender.passthru.tests.render

[the-furry-hubofeverything]

@ofborg eval

[the-furry-hubofeverything]

@ofborg build blender blender.passthru.tests.render

Turns out that's not what I wanted - I don't know how to cancel it, I apologize for taking up the resource

[the-furry-hubofeverything]

Dang it wrong PR

[amarshall]

There are more files than just scripts/addons absent in release tarball vs. Git. Most are irrelevant, but some are important. Notably release/datafiles/assets (from blender-assets repo). We at least need those to be viable.

diff -ur blender-4.1.0 blender-git

Only in blender-git/build_files/build_environment/patches: .gitattributes
Only in blender-git .git
Only in blender-git .git-blame-ignore-revs
Only in blender-git .gitignore
Only in blender-git .gitmodules
Only in blender-git lib
Only in blender-4.1.0/release/datafiles/assets: publish
Only in blender-4.1.0/release/datafiles/assets: README
Only in blender-4.1.0/release/datafiles/assets: testing
Only in blender-4.1.0/scripts: addons
Only in blender-git/tests: data

What’s the underlying rationale for switching here? As you say, it is a lot slower to download. The rationale in the PR is just “from what I understand, this is common practice”—in reality it varies quite a bit. In the previous PR it was “to find the tagged sources easier”—but I don’t really understand that. Certainly there is some value in using the repos directly (in light of xz), and this would theoretically make it possible to more easily use un-tagged prereleases from Git (in reality it’s a bit complex because the other repos have different lifecycles) (and I have ad-hoc changed to fetchFromGitea to test prereleases before—but I didn’t really care about addons or assets for that). Finally, Blender’s (tarball) build process is quite complex, and I worry about missing something in the future. To be clear: this isn’t me saying “this is a bad idea”, but rather: I’m not totally sure which direction is better.

As an aside: it would be really nice to have an updateScript, as there will be at least three hashes to manually update after this.

[the-furry-hubofeverything]

There are more files than just scripts/addons absent in release tarball vs. Git. Most are irrelevant, but some are important. Notably release/datafiles/assets (from blender-assets repo). We at least need those to be viable.

Turns out they are using submodules and LFS for assets, lib, and tests (blender commit, arch issue)

As fetchLFS isn't implemented on fetchFromGithub (and therefore fetchFromGitea), I'll change the implementation to fetchgit.

Finally, Blender’s (tarball) build process is quite complex, and I worry about missing something in the future. To be clear: this isn’t me saying “this is a bad idea”, but rather: I’m not totally sure which direction is better.

To be honest, I don't either, so I asked on the matrix server. nevivurn made a good point that if the release servers ever go down, it's easier to get the git repo rather than the specific tarball.

[the-furry-hubofeverything]

If we don't use LFS, I could just add the respositories needed - looking at arch pkgbuild, there's three additional repos to include.

[the-furry-hubofeverything]

Suggested change

	srcs = [
	(fetchgit {
	name = "source";
	url = "https://projects.blender.org/blender/blender.git";
	rev = "v${finalAttrs.version}";
	hash = "sha256-4LtBOufcaUJP49QxT9/EcFVjNbyvGf/gYFoVtljod1Y=";
	fetchLFS = true;
	})
	(fetchFromGitea {
	name = "addons";
	domain = "projects.blender.org";
	owner = "blender";
	repo = "blender-addons";
	rev = "v${finalAttrs.version}";
	hash = "sha256-OHEUtGiubZNf3nsT4LfHuGyOhfVPk2H3osZ1lWJL4jI=";
	})
	];
	sourceRoot = "source";
	prePatch = ''
	mkdir scripts/addons
	cp -Rv --no-preserve=all ../addons/* scripts/addons
	'';
	srcs = [
	(fetchFromGitea {
	inherit domain owner;
	repo = "blender";
	rev = "v${finalAttrs.version}";
	hash = "sha256-Adpn9ueRWHoH+PZLtH2m/HwulyGTkxVXF6jnEncO+Bk=";
	})
	(fetchFromGitea {
	name = "addons";
	inherit domain owner;
	repo = "blender-addons";
	rev = "v${finalAttrs.version}";
	hash = "sha256-OHEUtGiubZNf3nsT4LfHuGyOhfVPk2H3osZ1lWJL4jI=";
	})
	(fetchFromGitea {
	name = "addons-contrib";
	inherit domain owner;
	repo = "blender-addons-contrib";
	rev = "v${finalAttrs.version}";
	hash = "sha256-XbwtpMoD1GSbogQ7uIxtnoi1vRjOi6DvTTI7GqhlkSw=";
	})
	(fetchFromGitea {
	name = "assets";
	inherit domain owner;
	repo = "blender-assets";
	rev = "v${finalAttrs.version}";
	hash = "sha256-AkN+VpwPlWCXYtsfjzC99gqupzh96tT7VuTRQjkPBDg=";
	})
	];
	sourceRoot = "source";
	prePatch = ''
	mkdir scripts/addons scripts/addons_contrib
	cp -rv --no-preserve=all ../addons/* scripts/addons
	cp -rv --no-preserve=all ../addons-contrib/* scripts/addons_contrib
	cp -rv --no-preserve=all ../assets/working/* release/datafiles/assets
	'';

This is to grab all sources without the use of LFS (domain and owner is added in let block)

[amarshall]

It seems you’re conflating LFS and submodules?

[the-furry-hubofeverything]

Yeah, I am. I didn't really understand it before, but Blender now uses both.

as per https://projects.blender.org/blender/blender/commit/3dc832a90415635c8387ba641d7722a9fd1e236c, they switched from SVN to git submodules with LFS.

[the-furry-hubofeverything]

Again, this is just an alternative, if fetchgit isn't really desired.

[the-furry-hubofeverything]

Another advantage to this approach is that we can control which version of the addons, test etc. are included.