Placement group contains already 10 servers

[jonas-tm]

I was trying to add a new node agent pool, but Terraform is showing me the following error.

│ Error: placement group 122312 contains already 10 servers (service_error)
│
│ with module.kube-hetzner.module.agents["3-0-agent-large2"].hcloud_server.server,
│ on .terraform/modules/kube-hetzner/modules/host/main.tf line 22, in resource "hcloud_server" "server":
│ 22: resource "hcloud_server" "server" {

For placement group config placement_group_disable it says:

That may be useful if you need to deploy more than 500 nodes! The default is "false".

As I have much fewer nodes I was not expecting any problems. Do I have to enable this flag as soon as I have more than 10 nodes in the cluster?

[mysticaltech]

@jonas-tm Just comment out this flag, or set it explicitly to false, and it will automatically handle as many servers as you need.

[jonas-tm]

@mysticaltech Now I am bit confused.

Now if you turn it to true, it means it disables it, and then you cannot deploy more than 10 nodes, ...

Do you mean can instead of cannot ?
Because I did not set it to true so placement group spread is enabled in my HCLOUD project causing my max 10 nodes issue described.

[mysticaltech]

So I misunderstood you, please do share your kube.tf without sensitive values, I need to reproduce to understand what is happening. It shouldn't be the case.

[mysticaltech]

I just looked at the logic and may have identified an issue. Just provide me your kube.tf and will run some tests.

[jonas-tm]

Here is my config:

Epand me

locals {
  location           = "ash"
  network_zone       = "us-east"
  kubernetes_version = "v1.25"
  cluster_name       = "kube-hetzner"

  ssh_public_key  = file("../../mykey.pub")
  ssh_private_key = file("../../mykey")
}

provider "hcloud" {
  token = var.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}


module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token

  source  = "kube-hetzner/kube-hetzner/hcloud"
  version = "1.9.1"

  ssh_public_key = local.ssh_public_key
   ssh_private_key = local.ssh_private_key
  network_region = local.network_zone # change to `us-east` if location is ash

  control_plane_nodepools = [
    {
      name        = "control-plane-${local.location}-1",
      server_type = "cpx21",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=control",
        "size=small",
        "control_number=1"
      ],
      taints = [],
      count  = 1
    },
    {
      name        = "control-plane-${local.location}-2",
      server_type = "cpx21",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=control",
        "size=small",
        "control_number=2"
      ],
      taints = [],
      count  = 1
    },
    {
      name        = "control-plane-${local.location}-3",
      server_type = "cpx21",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=control",
        "size=small",
        "control_number=3"
      ],
      taints = [],
      count  = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx21",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=agent",
        "size=small"
      ],
      taints = [],
      count  = 3
    },
    {
      name        = "agent-large",
      server_type = "cpx41",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=4",
        "memory=8",
        "disk=160",
        "type=agent",
        "size=large"
      ],
      taints = [],
      count  = 6
    },
    {
      name        = "agent-large2",
      server_type = "cpx41",
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=4",
        "memory=8",
        "disk=160",
        "type=agent",
        "size=large"
      ],
      taints = [],
      count  = 6
    },
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = local.location

  autoscaler_nodepools = [
    {
      name        = "autoscaler-small"
      server_type = "cpx21", # must be same or better than the control_plane server type (regarding disk size)!
      location    = local.location,
      labels = [
        "location=${local.location}",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=autoscaler",
        "size=small"
      ],
      min_nodes = 0
      max_nodes = 15
    },
  ]
  
  ingress_controller = "nginx"

  kured_options = {
    "reboot-days" : "sa,su"
    "start-time" : "1am"
    "end-time" : "7am"
    "time-zone" : "UTC"
  }

  initial_k3s_channel = local.kubernetes_version

 
  cluster_name = local.cluster_name

  restrict_outbound_traffic = false

  use_control_plane_lb = true

    lb_hostname = "my-lb-domain.com"

  create_kubeconfig = false

}

[mysticaltech]

Thanks @jonas-tm, I will get back to you later today as soon as I have some time to focus on that.

[mysticaltech]

@jonas-tm Issues have been fixed in the newly released v1.10.4. By default placement group is enabled and you can have up to 500 nodes. If you disable it, you loose the physical spread of the machines but can deploy even more machines.

In both cases it will work now! 🚀

[jonas-tm]

Thanks, @mysticaltech really appreciate the work you put in!

[sshcherbinin]

release v1.10.4

for now this throws an error if there are 10 (20, 30, ...) nodes in agent_nodes

terraform:
The given key does not identify an element in this collection value: the given index is greater than or equal to the length of the collection.

for this:
hcloud_placement_group.agent[floor((index(keys(local.agent_nodes), each.key) + 1) / 10)].id

and this:
hcloud_placement_group.control_plane[floor((index(keys(local.control_plane_nodes), each.key) + 1) / 10)].id

floor((10 + 1) / 10) = 1

fix:
hcloud_placement_group.agent[floor((index(keys(local.agent_nodes), each.key)) / 10)].id
hcloud_placement_group.control_plane[floor((index(keys(local.control_plane_nodes), each.key)) / 10)].id

floor(0 / 10) = 0
...
floor(10 / 10) = 0
floor(11 / 10) = 1
...
floor(19 / 10) = 1
floor(20 / 10) = 2

[mysticaltech]

@sshcherbinin In your example you say floor(10 / 10) = 0, but indeed it's equal to 1. But your fix is correct, the + 1 needs to go. Please don't hesitate to push a PR. I will test and release ASAP.

[mysticaltech]

Published the fix in v1.10.5, thanks again @sshcherbinin, good catch!!