`--build-host` option in `nh` is misleading compared to `nixos-rebuild`

[c2fc2f]

The --build-host option in nh does not behave the same way as in nixos-rebuild, which makes it confusing and arguably useless in its current state.

Behavior in nh

When using --build-host, nh simply forwards the host to nix build --builders:

• nh/src/nixos.rs

  Line 195 in 8bf3234

  .builder(self.build_host.clone())

• nh/src/commands.rs

  Lines 664 to 666 in 8bf3234

  	Some(host) => {
  	vec!["--builders".to_string(), format!("ssh://{host} - - - 100")]
  	},

• nh/src/interface.rs

  Lines 813 to 815 in 8bf3234

  	if let Some(ref builders) = self.builders {
  	args.push("--builders".into());
  	args.push(builders.clone());

This means that the SSH connection is initiated by the nix-daemon (running as root), in a non-interactive environment.
As a consequence, if the SSH key used to connect to the builder requires a passphrase, it must already be unlocked and available in root’s SSH agent. Since nix-daemon does not have access to the user’s SSH agent, this often fails unless you use a passphrase-less key or preload it into an agent that root can access.

See also: nix.dev distributed builds

Behavior in nixos-rebuild

In contrast, nixos-rebuild’s --build-host works differently:

• It opens an SSH connection as the user running the command, in an interactive environment (so it has access to the user’s SSH agent, or the user can even type the passphrase if necessary).
• It then transfers the system closure to the build host via that connection and launches the build remotely.

Relevant code:

• https://github.com/NixOS/nixpkgs/blob/b71d5fb26af9c12377192522fc79f89e50554c22/pkgs/by-name/ni/nixos-rebuild-ng/src/nixos_rebuild/nix.py#L165-L177
• https://github.com/NixOS/nixpkgs/blob/b71d5fb26af9c12377192522fc79f89e50554c22/pkgs/by-name/ni/nixos-rebuild-ng/src/nixos_rebuild/process.py#L111-L122

Why this is problematic

• In nh, --build-host is effectively just an alias for --builders.
• This is misleading, because it gives the impression that it behaves like nixos-rebuild --build-host, when in reality it does not.
• For many users, this makes nh --build-host unusable unless they set up special conditions (e.g. SSH keys without passphrases available to root).

Conclusion

There are two possible ways forward:

1. Remove the --build-host option from nh to avoid confusion.
2. Reimplement it so that it matches the expected semantics from nixos-rebuild, i.e. opening the SSH connection as the user in an interactive environment, transferring the system closure, and building remotely.

Related discussions

• nh os switch --build-host <HOST> still builds locally #308 (comment)
• --builders does not use identities from ssh-agent when key has a passphrase NixOS/nix#14125 (comment)

Maintainers

@NotAShelf
@cramt

---

Add 👍 to issues you find important.

[NotAShelf]

I've submitted #497 to hopefully address this issue. It refactors the build logic from ground up, and seems to be working in my setup as it did before. Though I did not have any passphrases for my remote builder, so until I'm on my main system I'm not going to be able to test it thoroughly. In the meantime, if you (or any of the 14 people that +1'd this issue) could test it would be valuable in advancing my PR and getting it merged for the 4.3.0 release.

I've also added --build-host support for darwin and home platform commands, so testing would be great if anyone uses those platforms and has a suitable build host. I assume building Darwin with a NixOS builder will be difficult unless you have cross-builders but nh home should be easy to test if you have a suitable machine.

[c2fc2f]

--build-host Only

It seems to work correctly with --build-host:

❯ ssh-add -D
All identities removed.
❯ nh os switch --build-host sisyphe.sagbot.com
> Building NixOS configuration
> Evaluating derivation path
> Copying closure to build host 'sisyphe.sagbot.com'
Enter passphrase for key '/home/culottes/.ssh/culottes': 
Confirm user presence for key ED25519-SK SHA256:fzZ5jhDmnTGXzANnXic45pIejy1uKrWtWIXWj+ixqcA
User presence confirmed
copying 0 paths...
> Building on remote host 'sisyphe.sagbot.com'
Finished at 11:25:28 after 0s
> Copying result from build host 'sisyphe.sagbot.com'
copying 0 paths...
> Activating configuration
Please touch the FIDO authenticator.
[sudo] password for culottes: 
[ ... ]
> Adding configuration to bootloader

--build-host and --target-host

When I try to build the configuration on my server:

❯ nh os switch --target-host sisyphe.sagbot.com --build-host sisyphe.sagbot.com -H sisyphe
> Building NixOS configuration
> Evaluating derivation path
> Copying closure to build host 'sisyphe.sagbot.com'
[ ... ]
> Building on remote host 'sisyphe.sagbot.com'
[ ... ]
┏━ Dependency Graph:
┃                ┌─ ⏵ hm_fontconfigconf.d10hmfonts.conf ⏱ 1s
┃             ┌─ ⏵ home-manager-files ⏱ 1s
┃          ┌─ ⏵ home-manager-generation ⏱ 1s
┃       ┌─ ⏵ unit-home-manager-culottes.service ⏱ 1s
┃    ┌─ ⏵ system-units ⏱ 1s
┃    │  ┌─ ⏵ home-manager-path ⏱ 1s
┃    ├─ ⏵ user-environment ⏱ 1s
┃ ┌─ ⏵ etc
┃ ⏵ nixos-system-sisyphe-26.05.20251204.57fe540
┣━━━ Builds         
┗━ ∑ ⏵ 9 │ ✔ 0 │ ⏸ 0 │ Finished at 11:18:59 after 3s
> Copying closure from 'sisyphe.sagbot.com' to 'sisyphe.sagbot.com'
Error: 
   0: Target profile path does not exist: /tmp/nh-osoPi7Ji/result

Location:
   src/nixos.rs:437

When I try to build the configuration on my other server with my build server:

❯ nh os switch --target-host icare.sagbot.com --build-host sisyphe.sagbot.com -H icare
> Building NixOS configuration
> Evaluating derivation path
> Copying closure to build host 'sisyphe.sagbot.com'
[ ... ]
icare-26.05.20251204.57fe540.drv' to 'ssh://sisyphe.sagbot.com'...
> Building on remote host 'sisyphe.sagbot.com'
[ ... ]
┏━ Dependency Graph:
┃ ┌─ ⏵ boot.json
┃ │     ┌─ ⏵ unit-dbus.service ⏱ 1s
┃ │     │        ┌─ ⏵ hm_fontconfigconf.d10hmfonts.conf ⏱ 2s
┃ │     │     ┌─ ⏵ home-manager-files ⏱ 2s
┃ │     │  ┌─ ⏵ home-manager-generation ⏱ 1s
┃ │     ├─ ⏵ unit-home-manager-culottes.service ⏱ 1s
┃ │     │  ┌─ ⏵ X-Restart-Triggers-polkit ⏱ 2s
┃ │     ├─ ⏵ unit-polkit.service ⏱ 1s
┃ │     │  ┌─ ⏵ X-Restart-Triggers-systemd-tmpfiles-resetup ⏱ 3s
┃ │     ├─ ⏵ unit-systemd-tmpfiles-resetup.service ⏱ 2s
┃ │  ┌─ ⏵ system-units ⏱ 1s
┃ │  │     ┌─ ⏵ X-Restart-Triggers-dbus ⏱ 1s
┃ │  │  ┌─ ⏵ unit-dbus.service ⏱ 1s
┃ │  ├─ ⏵ user-units ⏱ 1s
┃ │  ├─ ⏵ dbus-1 ⏱ 2s
┃ │  │  ┌─ ⏵ home-manager-path ⏱ 3s
┃ │  ├─ ⏵ user-environment ⏱ 2s
┃ │  ├─ ⏵ user-generators ⏱ 3s
┃ │  ├─ ⏵ system-shutdown ⏱ 3s
┃ │  ├─ ⏵ system-generators ⏱ 3s
┃ │  ├─ ⏵ tmpfiles.d ⏱ 3s
┃ ├─ ⏵ etc
┃ │  ┌─ ⏵ closure-info ⏱ 1s
┃ │  ├─ ⏵ linux-6.12.60-modules-shrunk ⏱ 3s
┃ ├─ ⏵ initrd-linux-6.12.60 ⏱ 1s
┃ │  ┌─ ⏵ grub-config.xml ⏱ 3s
┃ ├─ ⏵ install-grub.sh ⏱ 2s
┃ ├─ ⏵ system-path ⏱ 3s
┃ ├─ ⏵ manifest.json ⏱ 3s
┃ ⏵ nixos-system-icare-26.05.20251204.57fe540
┣━━━ Builds           │ Downloads       │ Host                         
┃    ⏵ 30 │     │     │     │     │     │ localhost                    
┃         │     │     │     │ ↓ 1 │     │ https://cache.nixos.org      
┗━ ∑ ⏵ 30 │ ✔ 0 │ ⏸ 0 │ ↓ 0 │ ↓ 1 │ ⏸ 0 │ Finished at 11:20:18 after 5s
> Copying closure from 'sisyphe.sagbot.com' to 'icare.sagbot.com'
Error: 
   0: Target profile path does not exist: /tmp/nh-oso2ykvn/result

Location:
   src/nixos.rs:437

[NotAShelf]

I had not tried --target-host. Though I have a general idea why this is occuring. I'll tackle it later today alongside another issue that has been bothering me.

[NotAShelf]

I've updated the branch with my latest changes. This should hopefully resolve the issue and the other issue with invalid args being passed to NOM. Worth nothing that I have not been able to test the changes and I will not be work on this PR for the foreseeable future due to my increasing health concerns. I'm hopeful that it will work, but I cannot make any guarantees. Further comments, testing, and even patches would be appreciated.

[c2fc2f]

I ran some tests and it seems to work perfectly.

Although there are still a couple of things that could be improved, in my opinion. Firstly, when the build host and target host are the same, it still copies to my computer and then to the server (a kind of ping pong effect), which could be avoided. I think this is due to this function:

https://github.com/nix-community/nh/blob/1f912df32d849cfee1bfd694f1f4621c6c0fdb23/src/remote.rs#L381C1-L386C47

And more generally, I have the impression that it opens several SSH connections instead of reusing the existing one, but I'm not sure how to fix that.

Personal message for NotAShelf

You've done an excellent job, and you can be truly proud of what you've accomplished, whether on nh or even on the Nix ecosystem. I sincerely wish you a speedy recovery and good health. You definitely deserve a break. You have my full support <3

[zierf]

I always found updating a remote server with a local build quite practical.

This was my original command to update my server.

nixos-rebuild switch --flake ~/FLAKEPATH#HOSTNAME --target-host root@SERVERADDRESS

Trying my scenario again with the changes of the latest commit in PR #497.
(tried building first and then updating)

nix shell github:nix-community/nh/7d6d1ba71e501fcb589661008baa441241019974                                                                                                                                                                                              

nh os -v build --target-host root@SERVERADDRESS --hostname HOSTNAME ~/FLAKEPATH
nh os switch --ask --target-host root@SERVERADDRESS --hostname HOSTNAME ~/FLAKEPATH

The build and package uploading process seems to be working quite well. It's also using the correct SSH key from my Keychain / SSH agent.

Unfortunately, after building and then applying the configuration, it starts asking for the root user password.

? [sudo] password for root@SERVERADDRESS:

It doesn't need to do that. The SSH certificate allows login as the root user without further password input (after it has already been loaded by the agent). The built packages were also previously uploaded as the root user without any further input.

Due to additional unnecessary password prompts, it has been a while since I attempted the server update with nh.
However, after testing it again with my installed version, this seems to be in line with the current normal behavior.

---

So it looks like there is no obvious regression, at least in my scenario.

Thanks for the great work and I wish you all the best!

[c2fc2f]

It doesn't need to do that. The SSH certificate allows login as the root user without further password input (after it has already been loaded by the agent). The built packages were also previously uploaded as the root user without any further input.

Yes, it's true that it always asks for the password even if you're logged in as root. It comes from here:

nh/src/commands.rs

Lines 510 to 528 in ef8d8c4

	let sudo_password = if self.ssh.is_some() && self.elevate.is_some() {
	let host = self.ssh.as_ref().ok_or_else(|| {
	eyre::eyre!("SSH host is None but elevation is required")
	})?;
	if let Some(cached_password) = get_cached_password(host) {
	Some(cached_password)
	} else {
	let password =
	inquire::Password::new(&format!("[sudo] password for {host}:"))
	.without_confirmation()
	.prompt()
	.context("Failed to read sudo password")?;
	let secret_password = SecretString::new(password.into());
	cache_password(host, secret_password.clone());
	Some(secret_password)
	}
	} else {
	None
	};

Specifically, this line:

self.elevate.is_some()

Because from what I've seen in the code, self.elevate is only None if nh is launched with --bypass-root-check, but here we need to check if the ssh login user is root

nh/src/nixos.rs

Line 234 in ef8d8c4

.elevate(elevate.then_some(elevation.clone()))

nh/src/nixos.rs

Line 296 in ef8d8c4

let elevate = has_elevation_status(self.bypass_root_check)?;

nh/src/interface.rs

Lines 230 to 232 in ef8d8c4

	/// Don't panic if calling nh as root
	#[arg(short = 'R', long, env = "NH_BYPASS_ROOT_CHECK")]
	pub bypass_root_check: bool,

---

And then the password, if necessary, will be used to launch the command with elevation if necessary.

nh/src/commands.rs

Lines 581 to 589 in ef8d8c4

	let cmd = ssh_wrap(
	if self.show_output {
	cmd.stderr(Redirection::Merge)
	} else {
	cmd.stderr(Redirection::None).stdout(Redirection::None)
	},
	self.ssh.as_deref(),
	sudo_password.as_ref(),
	);

---

It should also be noted that if there is no elevation, then the command launch is not correct, since the environment is not evaluated on the remote machine (via SSH) but on the local machine.

nh/src/commands.rs

Line 577 in ef8d8c4

self.apply_env_to_exec(Exec::cmd(&self.command).args(&self.args))

[c2fc2f]

Also, nixos-rebuild only asks for the sudo password of the target host if --ask-sudo-password is passed as an argument.

--ask-sudo-password, -S
   When set, nixos-rebuild will ask for sudo password for remote activation (i.e.: on --target-host) at the start of the build process. Implies --sudo.