Ensure git exists in Steam's FHSEnv/Various fixes and additions for the Flake

[DrymarchonShaun]

So, originally this was just going to be a PR to add git to steam's extraPkgs because its not included by default and if the user doesn't have it installed (or steam is launched from somewhere that doesn't have git included in the PATH, like a systemd service), GitPython freaks out because it can't find the git executable and Millennium's core plugin fails to load.

As I was adding that, a few other small things jumped out at me, (ie, the pnpmDeps hash for the SDK was out of date since the sdk has been updated) so I've included them here to avoid spamming PRs.

I also saw the TODO for automating the pnpmDeps hashes, I'm not sure the best/easiest way to fully automate it, but I put together a script that should update both hashes just by running nix run .#update-pnpm-hashes (although nix-update does seem rather inconsistent so YMMV.)

Each of the various changes has its own commit, if anything needs to be changed or removed LMK.

[DrymarchonShaun]

Can't request a review but @Sk7Str1p3, it seems you're the maintainer for the flake?

[shdwmtr]

No @Sk7Str1p3 is the sole maintainer of all things nix. We need more nix maintainers.

[DrymarchonShaun]

@Neubulae when you get a chance, can you test this PR as well and see if it's working for you?

[DrymarchonShaun]

@Trivaris oh, I figured out what the problem is, your hashes are correct if the flake.lock in this repo is updated (see the 2 commits above). You must have missed a commit updating the flake in your fork when making your PR. I think this contains everything from both PRs now, once we get confirmations that everything in this PR is working I think it should be good to be merged @shdwmtr.

[Neubulae]

@Neubulae when you get a chance, can you test this PR as well and see if it's working for you?

I put this in my flake:

millennium = {
      url = "git+https://github.com/DrymarchonShaun/Millennium.git?ref=nix-add-git-dep&shallow=1";
};

and then built.
Still got hash mismatch:

error: hash mismatch in fixed-output derivation '/nix/store/g6ldv87lnp0ckyml7s8p1h4xl1zxga1w-millennium-assets-pnpm-deps.drv':
           likely URL: (unknown)
            specified: sha256-/H3Np/FjxEdU/TwqPPJlpNti2vfyNqUQ2CNIvzlSemA=
                  got: sha256-nDSltpFQRM9loVuDour4OrRdN22/A7MkZTGAtL0x7rU=
        expected path: /nix/store/9zb9wi9a3cqjnj51723bxcka5ndcyxzf-millennium-assets-pnpm-deps
             got path: /nix/store/znh3m89jq1qmfz4jq84b0gzdqm46ic2j-millennium-assets-pnpm-deps

[DrymarchonShaun]

Still got hash mismatch:

How about after running nix flake update millennium?

EDIT- You also don't have millennium.inputs.nixpkgs.follows = "nixpkgs"?

[Neubulae]

Still got hash mismatch:

How about after running nix flake update millennium?

I just git cloned and nix builded, looks like it's working fine.

I shall try updating.
EDIT: Updated, and things worked fine.

EDIT- You also don't have millennium.inputs.nixpkgs.follows = "nixpkgs"?

I do not.

[Neubulae]

It took me quite a few days to realize I cannot find Millenium's settings in my menu, even tho Millenium seems injected judging from the logs.
I notice this happening:

[34:14.638] ERROR couldn't get thread state ptr from plugin [core], maybe it crashed or exited early?
[34:14.638] * FUNCTION: Python::EvalResult Python::LockGILAndInvokeMethod(std::string, nlohmann::json_abi_v3_11_3::json)
[34:14.638] * LOCATION: https://github.com/SteamClientHomebrew/Millennium/blob//src/core/_c_py_interop.cc#L431

is this normal?

[Trivaris]

Oh I didnt notice the conversation continued in this pr, please let me know if I should test anything!

[DrymarchonShaun]

Oh I didnt notice the conversation continued in this pr, please let me know if I should test anything!

If you can verify that everything builds and functions for you, that would be great.

[shdwmtr]

does anyone know anything about

flake.lock
flake.nix
shell.nix
env.rc

If so do they need to be in the root directory of the repository?

[DrymarchonShaun]

flake.lock and flake.nix do, yes. Technically .envrc does too, but I don't know that's it's really necessary for this repo. It is possible to move shell.nix into the nix/ folder.

[shdwmtr]

Ah okay, because I'm in the progress of re-organizing the repo in this branch
https://github.com/SteamClientHomebrew/Millennium/tree/python-to-lua

I imagine it needs fixing though as I've just thrown the nix stuff in /pkgs/nix.

Nothing annoys me more than a repo with so many top level files you have to scroll noticeably far to get to the README

[shdwmtr]

Apparentally there is a nix build github:user/repo?dir=some-subfolder syntax? No idea if that is applicable though

[DrymarchonShaun]

Apparentally there is a nix build github:user/repo?dir=some-subfolder syntax? No idea if that is applicable though

There might be, I've never seen that used anywhere though. pkgs/nix/ should work as you have it, the paths in flake.nix will need to be updated though.

[Trivaris]

yes, that syntax does exist, for example
nix flake show gitlab:rycee/nur-expressions?dir=pkgs/firefox-addons

that syntax is basically just shorthand for https addresses and can be used for inputs in other flakes, so no problem there

[Trivaris]

Another problem that I've just noticed in flake.nix is

python = {
  millennium = pkgs.callPackage ./nix/python/millennium.nix { };
  core-utils = pkgs.callPackage ./nix/python/core-utils.nix { };
};

Packages in flake outputs always have to be top level attributes. I suggest changing them to something like "python/millennium" = pkgs.callPackage ... at some point

[Trivaris]

I'm currently experimenting with migrating the git submodules to flake inputs and it's going well so far
@shdwmtr is nix supposed to be in addition to other tools, or is it in any other way necessary to keep sdk, crow, etc as submodules?

[Sk7Str1p3]

Packages in flake outputs always have to be top level attributes

This is optional. Many packages in nixpkgs are part of attrSet, for example winePackages

[Sk7Str1p3]

does anyone know anything about

flake.lock
flake.nix
shell.nix
env.rc

If so do they need to be in the root directory of the repository?

Yes, flake.nix is root of flake. Lockfile pins inputs commits. Shell.nix is development shell so contributors can develop without installing dependencies globally; .envrc is used by direnv to launch devshell automatically, it also integrates with vscode and other IDEs

[Trivaris]

Packages in flake outputs always have to be top level attributes

This is optional. Many packages in nixpkgs are part of attrSet, for example winePackages

So I did a little bit of reading and it turns out that you need to signal to the flake that that is an attrset with

python = lib.recurseIntoAttrs {
  millennium = pkgs.callPackage ./nix/python/millennium.nix { };
  core-utils = pkgs.callPackage ./nix/python/core-utils.nix { };
};

else nix flake check errors out, which is how I got aware of of this in the first place

EDIT: Its not the usage of lib.recurseIntoAttrs, but the usage of outputs.legacyPackages instead of the outputs.packages, which allows nixpkgs to have nested packages.
However this would turn the build command from
nix build .#millennium-python
to
nix build .#legacyPackages.x86_64-linux.python.millennium
which is why I propose we do move to my suggestion

[Trivaris]

I opened #499 in which I changed the usage of relative paths to the respective git submodules into usage of flake inputs, which should provide a cleaner way of managing versions.
This also shouldn't impact any other compiling methods, as they remain unchanged and the submodules do still exist.

[Sk7Str1p3]

However this would turn the build command from nix build .#millennium-python to nix build .#legacyPackages.x86_64-linux.python.millennium

Are you sure? AFAIK you can build nixpkgs package (which is part of legacyPackages) with nix build github:nixos/nixpkgs#hello, without writing full 'path'. LMK if I'm wrong

[shdwmtr]

So whats the status of this?

[DrymarchonShaun]

AFAIK everything here is working. IIRC I was waiting for @Trivaris to verify it was working for them before i considered it ready to merge.

[Neubulae]

It took me quite a few days to realize I cannot find Millenium's settings in my menu, even tho Millenium seems injected judging from the logs. I notice this happening:

[34:14.638] ERROR couldn't get thread state ptr from plugin [core], maybe it crashed or exited early?
[34:14.638] * FUNCTION: Python::EvalResult Python::LockGILAndInvokeMethod(std::string, nlohmann::json_abi_v3_11_3::json)
[34:14.638] * LOCATION: https://github.com/SteamClientHomebrew/Millennium/blob//src/core/_c_py_interop.cc#L431

is this normal?

my Millennium still doesn't show up in the menu, does anybody have this issue or it's due to this PR's Millennium being an older version?

[shdwmtr]

@Neubulae

The error is likely caused by a fail earlier in the log

[Neubulae]

@Neubulae

The error is likely caused by a fail earlier in the log

Do I have ways to extract something more detailed? Plus my console is spammed by libmillennium.so elf related errors which might be a visual clutter

[DrymarchonShaun]

Do I have ways to extract something more detailed? Plus my console is spammed by libmillennium.so elf related errors which might be a visual clutter

Usually I just keep a terminal window open with tail -f ~/.local/share/Steam/logs/console-linux.txt running, then launch steam as usual (after making sure It's not already running)

you can also pipe into grep like tail -f ~/.local/share/Steam/logs/console-linux.txt | grep -v "wrong ELF class: ELFCLASS32" to hide the ELF error spam

[Neubulae]

Do I have ways to extract something more detailed? Plus my console is spammed by libmillennium.so elf related errors which might be a visual clutter

Usually I just keep a terminal window open with tail -f ~/.local/share/Steam/logs/console-linux.txt running, then launch steam as usual (after making sure It's not already running)

you can also pipe into grep like tail -f ~/.local/share/Steam/logs/console-linux.txt | grep -v "wrong ELF class: ELFCLASS32" to hide the ELF error spam

I tried looking into the log and found it complaining about python3.11 not being found. Only to realize later that it's been linked to a nix store path that no longer exists.
After rebuilding the .venv it's looking alright.