Remote exec doesn't work with ssh agent

[pschiffe]

Description

When trying to use ssh key agent to connect to the vms, terraform fails to connect.
The first part, the ssh check works:

until ssh ${local.ssh_args} -i /tmp/${random_string.identity_file.id} -o ConnectTimeout=2 -p ${var.ssh_port} root@${self.ipv4_address} true 2> /dev/null

But the remote exec itself doesn't, due to the connection setting agent_identity. That option expects path to the pub file, not it's content:

module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Host: 168.xx.179
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Private key: false
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server (remote-exec):   Target Platform: unix
2023-04-15T00:34:28.788+0200 [DEBUG] Connecting to 168.xx.179:2222 for SSH
2023-04-15T00:34:28.806+0200 [DEBUG] Connection established. Handshaking for user root
2023-04-15T00:34:28.953+0200 [DEBUG] error reading "/home/pschiffe/terraform/hetzner/ssh-ed25519 AAAACxx6grpncD key\n": open /home/pschiffe/terraform/hetzner/ssh-ed25519 AAAACxx6grpncD key
: no such file or directory
2023-04-15T00:34:28.953+0200 [DEBUG] error reading "/home/pschiffe/terraform/hetzner/ssh-ed25519 AAAACxx6grpncD key\n.pub": open /home/pschiffe/terraform/hetzner/ssh-ed25519 AAAACxx6grpncD key
.pub: no such file or directory
2023-04-15T00:34:29.000+0200 [WARN]  SSH authentication failed (root@168.xx.179:2222): ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures
2023-04-15T00:34:29.000+0200 [WARN]  retryable error: SSH authentication failed (root@168.xx.179:2222): ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures
2023-04-15T00:34:29.000+0200 [INFO]  sleeping for 20s

Kube.tf file

# Customize the SSH port (by default 22)
  ssh_port = 2222

  # * Your ssh public key
  ssh_public_key = file("~/.ssh/hetzner_id_ed25519.pub")
  # * Your private key must be "ssh_private_key = null" when you want to use ssh-agent for a Yubikey-like device authentification or an SSH key-pair with a passphrase.
  # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md
  ssh_private_key = null
  # You can add additional SSH public Keys to grant other team members root access to your cluster nodes.
  # ssh_additional_public_keys = []

Screenshots

No response

Platform

Linux

[mysticaltech]

@pschiffe Your public key path is wrong, it should be /home/pschiffe/terraform/hetzner/ssh-ed25519.pub otherwise you need to change your ssh-agent to use ~/.ssh/hetzner_id_ed25519 as private key.

So this is probably not a bug, you have a mismatch between your public key and the private key loaded through your ssh-agent.

[mysticaltech]

Also FYI, you need to cleanupkh and retry, as your IP may have been blacklisted by the nodes by now.

[pschiffe]

My private key is only in ssh key agent, not on the file system (it's loaded from keepassxc). I've created the file ~/.ssh/hetzner_id_ed25519.pub with public key which matches the private key in the ssh agent. I can log in to ssh with option -i ~/.ssh/hetzner_id_ed25519.pub.

Now, here's the content of the file ~/.ssh/hetzner_id_ed25519.pub read and stored in the temp file, which is then used with -i option, and that's fine and works: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/modules/host/main.tf#L58

Problem is here, where it's still the content of the file, not the file path: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/modules/host/main.tf#L49

If I use ssh_public_key = "~/.ssh/hetzner_id_ed25519.pub" in kube.tf, remote exec should work, but the ssh check won't.

[pschiffe]

The ssh-ed25519 AAAACxx6grpncD key\n part in the DEBUG tf log in my original post is the content of the ~/.ssh/hetzner_id_ed25519.pub file, sorry if that wasn't clear.

Thanks for the note about blacklisting, I'm aware.

[mysticaltech]

@pschiffe How of curiosity, do try by commenting out the ssh_port option. You can always close the firewall ssh port when the cluster is deployed. (The reason I say this is that I am not sure that custom ports will work with ssh agents the way we have it setup).

Also make sure to triple check by following https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md

[pschiffe]

Hi @mysticaltech, thanks for looking into this issue.

I've tried with commenting out the ssh_port option, but it still doesn't work. It seems that the port is properly propagated to the connections settings of remote exec:
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/modules/host/main.tf#L51

When looking into the source code of terraform, I see that the agent_identity indeed doesn't support a full string of public key as an input, functions idKeyData() and findIDPublicKey() here:
https://github.com/hashicorp/terraform/blob/main/internal/communicator/ssh/provisioner.go#L478

But there's this one interesting condition in here:
https://github.com/hashicorp/terraform/blob/main/internal/communicator/ssh/provisioner.go#L561
which basically says, that agent_identity option can contain the comment of the public key and it will find it in the agent. I've tested this and it works, in my case: agent_identity = key, based on the logs in the original post.

---

Now, there is also this thing. If terraform doesn't find the file defined in agent_identity, it just goes through all keys in the ssh agent. So if you have only 1 or 2 keys stored in the ssh agent, agent_identity = whatever will work. But if you have more than 2, it fails due to the MaxAuthTries 2 setting here:
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/locals.tf#L536

I have more than 3 keys in my ssh agent, so it doesn't work for me. If I increase MaxAuthTries on the vm, then it works.

---

Here's my minimal example I'm testing this with:

resource "null_resource" "ssh-connection" {
  provisioner "remote-exec" {
    inline = [
      "uname -a"
    ]

    connection {
      host           = "49.xx.88"
      user           = "root"
      private_key    = null
      agent_identity = "key"
      port           = 2222
    }
  }
}

[mysticaltech]

@pschiffe Great debug! Basically, if I follow you correctly, if we parametrize MaxAuthTries via a variable, your problem can be fixed? Please PR welcome to add this variable, you could call it ssh_max_auth_tries for instance.

[pschiffe]

hi @mysticaltech,

parametrizing MaxAuthTries should work, even though it's only a workaround. However, I can imagine it being useful also in some other cases.

The culprit is that the agent_identity option is not set correctly. Ideally, I would drop dealing with ssh keys as strings, and just use them as files. That would solve this problem, and make the code a little bit simpler, as it currently has to create and delete temp files. Problem is this would break backwards compatibility. Also I see the commit that introduced it, but without explanation, here ed9658a

Alternatively, non-breaking change would be to set the local ssh_agent_identity variable to the key comment, ie:

  ssh_public_key_list = split(" ", var.ssh_public_key)
  ssh_public_key_comment = local.ssh_public_key_list[length(local.ssh_public_key_list) - 1]
  ssh_agent_identity = var.ssh_private_key == null ? local.ssh_public_key_comment : null

The third option could just be to expose the local ssh_agent_identity variable outside to kube.tf.

Please let me know what you think about these options. In the meantime I'll see if I can prepare the PR for parametrization of MaxAuthTries.

[mysticaltech]

@pschiffe Thanks for looking deeper into this. Please don't worry about breaking changes, this project is still young and users that use ssh-agent can fork the module and work around breaking changes to upgrade. Just please propose changes you think would be good! Usually, the simpler the better! 🙏

About the temp file reasoning, here it is in the description of that PR: #204

Just FYI, people that submitted those changes are no longer involved with the project.

[mysticaltech]

@pschiffe Any updates on this, were you able to make it work?

[pschiffe]

sorry @mysticaltech, I'm still on it, will try to prepare the PR soon

[pschiffe]

hi @mysticaltech, I've finally prepared the PR #774 , sorry it took some time.
So, I've played with it a little bit, and basically the remote exec connections from terraform could be fixed by adjusting the agent_identity property. However, there are at least two resources - data "remote_file" "kubeconfig" and data "remote_file" "kustomization_backup" where the conn cannot be configured in a way where you can select specific key from ssh key agent.
In conclusion, adjusting the MaxAuthTries ssh configuration is the only way how to make it work for me, and for people with multiple keys in ssh key agent (without getting rid of the remote_file data resource).

[pschiffe]

Fix released in https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/releases/tag/v2.1.4 by introducing new ssh_max_auth_tries variable.