Provisioner errors when deploying a new cluster #653

tbabut · 2023-03-15T10:57:40Z

tbabut
Mar 15, 2023

I am just learning and starting with deploying my first ever own cluster but failing with the following errors:

[...]
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentitiesOnly yes' -o PubkeyAuthentication=yes -i /tmp/kw4kx76l1u7rghtllee1 -o ConnectTimeout=2 -p 22 root@91.107.xxx.xxx true 2> /dev/null\ndo\n  echo \"Waiting for MicroOS to reboot and become available...\"\n  sleep 3\ndone\n"]
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Host: 91.107.xxx.xxx
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec): Connected!
module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server (remote-exec): /tmp/terraform_1656013099.sh: 3: Syntax error: redirection unexpected
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server: Still creating... [3m0s elapsed]
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server: Still creating... [3m10s elapsed]
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (local-exec): ssh: connect to host 167.235.xxx.xxx port 2233: Operation timed out
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentitiesOnly yes' -o PubkeyAuthentication=yes -i /tmp/fwlx06v9snmxkgohybtw -o ConnectTimeout=2 -p 22 root@167.235.xxx.xxx true 2> /dev/null\ndo\n  echo \"Waiting for MicroOS to reboot and become available...\"\n  sleep 3\ndone\n"]
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Host: 167.235.xxx.xxx
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec): Connected!
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server: Still creating... [3m20s elapsed]
module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server (remote-exec): /tmp/terraform_224137673.sh: 3: Syntax error: redirection unexpected
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 103, in resource "hcloud_server" "server":
│  103:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_678732453.sh": Process exited with status 2
╵
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.module.agents["1-0-agent-large"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 103, in resource "hcloud_server" "server":
│  103:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_224137673.sh": Process exited with status 2
╵
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.module.agents["2-0-storage"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 103, in resource "hcloud_server" "server":
│  103:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_1656013099.sh": Process exited with status 2
╵
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 103, in resource "hcloud_server" "server":
│  103:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_1745654389.sh": Process exited with status 2
╵
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.module.agents["3-0-egress"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 103, in resource "hcloud_server" "server":
│  103:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_222658456.sh": Process exited with status 2
╵

Here is my kube config file:

locals {
  hcloud_token = "xxxxxxxxxxxxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("~/k3s/ssh-k3s.pub")
  ssh_private_key = file("~/k3s/ssh-k3s")
  ssh_hcloud_key_label = "role=admin"
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx21",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "storage",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 1
    },
    {
      name        = "egress",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 1
    }
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  ingress_replica_count = 1

  enable_rancher = true
  rancher_hostname = "mycluster.somewhere.com"
  rancher_bootstrap_password = "xxxxxxxxxxxxxx"

}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

I have reduced the default amount of the nodes because of the early IP and resources limit of my new Hetzner account. I have also destroyed the cluster with the terraform command and tried again, but with no success.

Configuration seems to be valid:

$ terraform init --upgrade                                                                              
Initializing the backend...
Upgrading modules...
Downloading registry.terraform.io/kube-hetzner/kube-hetzner/hcloud 1.10.3 for kube-hetzner...
- kube-hetzner in .terraform/modules/kube-hetzner
- kube-hetzner.agents in .terraform/modules/kube-hetzner/modules/host
- kube-hetzner.control_planes in .terraform/modules/kube-hetzner/modules/host

Initializing provider plugins...
- Finding tenstad/remote versions matching ">= 0.0.23"...
- Finding integrations/github versions matching ">= 4.0.0"...
- Finding hashicorp/local versions matching ">= 2.0.0"...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/cloudinit...
- Finding hetznercloud/hcloud versions matching ">= 1.35.2"...
- Using previously-installed hashicorp/local v2.4.0
- Using previously-installed hashicorp/null v3.2.1
- Using previously-installed hashicorp/random v3.4.3
- Using previously-installed hashicorp/cloudinit v2.3.2
- Using previously-installed hetznercloud/hcloud v1.36.2
- Using previously-installed tenstad/remote v0.1.1
- Using previously-installed integrations/github v5.18.3

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform validate
Success! The configuration is valid.

The servers seem to be all in rescue mode at this stage. Thank you very much.

Answered by tbabut

Mar 16, 2023

Not quite right. I hadn't changed the ssh_port in the kube.tf file. I have had set a different standard port in my ssh config:

$ cat ~/.ssh/config
Host *
	ServerAliveInterval 60
	User tbabut
	Port 9962
[...]

During the deployment the following command is one of the steps, that have failed:

module.kube-hetzner.module.agents["3-0-egress"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentitiesOnly yes' -o PubkeyAuthentication=yes -i /tmp/bqvn2dunisjjbjbstoke root@5.75.xxx.xxx '(sleep 2; reboot)&'; sleep 3\n"]

The ssh command from above is including my ssh config (~/.ssh/config) with my changed port for a…

View full answer

tbabut · 2023-03-15T11:29:51Z

tbabut
Mar 15, 2023
Author

The rescue system is running Debian. But the failing script on the node is assuming an openSUSE system I guess:

root@rescue ~ # lsb_release -d
Description:	Debian GNU/Linux 11 (bullseye)

root@rescue ~ # bash -xv /tmp/terraform_222658456.sh
#!/bin/sh
set -ex
+ set -ex
transactional-update shell <<< "zypper --no-gpg-checks --non-interactive install https://github.com/k3s-io/k3s-selinux/releases/download/v1.3.testing.4/k3s-selinux-1.3-4.sle.noarch.rpm"
+ transactional-update shell
/tmp/terraform_222658456.sh: line 3: transactional-update: command not found

Or maybe the node shouldn't be in rescue mode at this stage.

I've also tried to use the latest terraform provider hcloud v1.36.2 without a difference.

0 replies

mysticaltech · 2023-03-15T15:52:56Z

mysticaltech
Mar 15, 2023
Maintainer

@tbabut You need to destroy everything properly and try again, it's super useful to use the hcloud cli to check things up and even delete hanged ressources. Also, you can shoot a support request to hetzner to up the limits.

0 replies

tbabut · 2023-03-15T16:57:55Z

tbabut
Mar 15, 2023
Author

Thank you for your response. I have destroyed everything and started over again multiple times. I have also checked via UI and hcloud-cli, wether everything is really vanished after running the terraform command. Now two hours later I tried again, but it failed with the same errors. Somehow the nodes are in rescue mode in this stage, but they aren't supposed to be, right? If I reboot the servers, they come up fine with openSUSE respectively MicroOS. But the deployment is failed of course.

I don't think the limits of my account at Hetzner is a problem right now. Unfortunately I cannot contact support at Hetzner to raise the limits, because my account too fresh. ;)

0 replies

tbabut · 2023-03-15T19:01:03Z

tbabut
Mar 15, 2023
Author

It is my fault, I am so sorry. 🙈 I have a different ssh default port in my ssh config, so the ssh commands from my machine didn't work properly – the remote host therefore couldn't be reached. After setting the standard ssh port back to 22 everything works now as expected. 👍

Maybe this is something to consider. Adding the port to the ssh command from the config file or enforcing it to port 22 would remove this pitfall in general.

Thank you all for the great project and contributions by the way.

0 replies

mysticaltech · 2023-03-15T21:32:48Z

mysticaltech
Mar 15, 2023
Maintainer

@tbabut So if I understood correctly, setting ssh_port to any value other than 22 causes the deploy to fail?

1 reply

mysticaltech Mar 15, 2023
Maintainer

(It shouldn't be the case, so it must be a bug)

tbabut · 2023-03-16T06:59:53Z

tbabut
Mar 16, 2023
Author

Not quite right. I hadn't changed the ssh_port in the kube.tf file. I have had set a different standard port in my ssh config:

$ cat ~/.ssh/config
Host *
	ServerAliveInterval 60
	User tbabut
	Port 9962
[...]

During the deployment the following command is one of the steps, that have failed:

module.kube-hetzner.module.agents["3-0-egress"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 'IdentitiesOnly yes' -o PubkeyAuthentication=yes -i /tmp/bqvn2dunisjjbjbstoke root@5.75.xxx.xxx '(sleep 2; reboot)&'; sleep 3\n"]

The ssh command from above is including my ssh config (~/.ssh/config) with my changed port for all hosts while the ssh daemon in the rescue system is listening to the default port 22. After removing the port 9962 from my config the next deployment ran like a breeze.

So I guess the ssh port 22 should be hardcoded for using the rescue system at least, because the user might have a different default setting in his config.

1 reply

mysticaltech Mar 17, 2023
Maintainer

Ok, great, it's already hard-coded! Thanks for the confirmation.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provisioner errors when deploying a new cluster #653

{{title}}

Replies: 6 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Provisioner errors when deploying a new cluster #653

tbabut Mar 15, 2023

Replies: 6 comments · 2 replies

tbabut Mar 15, 2023 Author

mysticaltech Mar 15, 2023 Maintainer

tbabut Mar 15, 2023 Author

tbabut Mar 15, 2023 Author

mysticaltech Mar 15, 2023 Maintainer

mysticaltech Mar 15, 2023 Maintainer

tbabut Mar 16, 2023 Author

mysticaltech Mar 17, 2023 Maintainer

tbabut
Mar 15, 2023

Replies: 6 comments 2 replies

tbabut
Mar 15, 2023
Author

mysticaltech
Mar 15, 2023
Maintainer

tbabut
Mar 15, 2023
Author

tbabut
Mar 15, 2023
Author

mysticaltech
Mar 15, 2023
Maintainer

mysticaltech Mar 15, 2023
Maintainer

tbabut
Mar 16, 2023
Author

mysticaltech Mar 17, 2023
Maintainer