cluster template for vsphere #1129

jkremser · 2023-10-13T10:13:00Z

What does this PR do?

Updating the old templates that don't work with the current capv provider to new ones that use cluster app instead. This is needed for auto IP allocation for workload cluster's api server. We use the ipam controller for that and IPAddressClaim CR is created by cluster helm chart.

./kubectl-gs template cluster --provider vsphere \
   --organization giantswarm \
   --name gstempl1 \
   --vsphere-service-load-balancer-cidr 10.10.222.238/32 | k apply -f -


# 3 minutes later..

λ k get vspheremachine -A | grep gstempl                                                                                                                                           
org-giantswarm   gstempl1-control-plane-05c119fe-f5tgr     gstempl1                                                              47s
org-giantswarm   gstempl1-control-plane-05c119fe-jlrxz     gstempl1     true    vsphere://4208a5f0-9f06-1c6a-c027-fb604dc6c388   3m37s
org-giantswarm   gstempl1-default-4418a143-9ttrk           gstempl1     true    vsphere://4208d914-3acc-5d7f-3fcd-247947a569f7   3m41s
org-giantswarm   gstempl1-default-4418a143-bdqq8           gstempl1     true    vsphere://4208ad8b-99ab-9334-ff8e-c63f9320d934   3m40s
org-giantswarm   gstempl1-default-4418a143-tmqpz           gstempl1     true    vsphere://42084f88-c4c0-7891-ecbd-52d27c529068   3m40s

λ govc ls /Datacenter/vm/ | grep gstempl                                                                                                                                                                   

/Datacenter/vm/gstempl1-gw4j5
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-tm2k7
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-gvprp
/Datacenter/vm/gstempl1-ts49x
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-vh9mj

# 7 minutes later..

λ k tree clusters -n org-giantswarm gstempl1  
NAMESPACE       NAME                                                           READY  REASON  AGE
org-giantswarm  Cluster/gstempl1                                               True           7m1s
org-giantswarm  ├─AdmissionReport/152bdc98-61f5-4801-b0e4-cc1232818642         -              7m1s
org-giantswarm  ├─BackgroundScanReport/152bdc98-61f5-4801-b0e4-cc1232818642    -              6m30s
org-giantswarm  ├─KubeadmConfigTemplate/gstempl1-worker-49eeb741               -              7m1s
org-giantswarm  ├─KubeadmControlPlane/gstempl1                                 True           7m1s
org-giantswarm  │ ├─Machine/gstempl1-2pqt2                                     True           86s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-kjc8p                             True           86s
org-giantswarm  │ │ │ └─Secret/gstempl1-kjc8p                                  -              86s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-845gn     True           86s
org-giantswarm  │ │   └─VSphereVM/gstempl1-2pqt2                               True           85s
org-giantswarm  │ ├─Machine/gstempl1-gw4j5                                     True           3m46s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-nwjff                             True           3m46s
org-giantswarm  │ │ │ └─Secret/gstempl1-nwjff                                  -              3m46s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-f5tgr     True           3m46s
org-giantswarm  │ │   └─VSphereVM/gstempl1-gw4j5                               True           3m46s
org-giantswarm  │ ├─Machine/gstempl1-ts49x                                     True           6m36s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-8jl25                             True           6m36s
org-giantswarm  │ │ │ └─Secret/gstempl1-8jl25                                  -              6m36s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-jlrxz     True           6m36s
org-giantswarm  │ │   └─VSphereVM/gstempl1-ts49x                               True           6m35s
org-giantswarm  │ ├─Secret/gstempl1-ca                                         -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-etcd                                       -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-kubeconfig                                 -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-proxy                                      -              6m36s
org-giantswarm  │ └─Secret/gstempl1-sa                                         -              6m36s
org-giantswarm  ├─MachineDeployment/gstempl1-worker                            True           7m1s
org-giantswarm  │ ├─AdmissionReport/da264148-fc21-42ba-8124-155d0a6acc81       -              7m1s
org-giantswarm  │ ├─BackgroundScanReport/da264148-fc21-42ba-8124-155d0a6acc81  -              6m29s
org-giantswarm  │ └─MachineSet/gstempl1-worker-84d565588xb6n95                 True           6m40s
org-giantswarm  │   ├─Machine/gstempl1-worker-84d565588xb6n95-gvprp            True           6m39s
org-giantswarm  │   │ ├─KubeadmConfig/gstempl1-worker-49eeb741-cmz7b           True           6m39s
org-giantswarm  │   │ │ └─Secret/gstempl1-worker-49eeb741-cmz7b                -              5m43s
org-giantswarm  │   │ └─VSphereMachine/gstempl1-default-4418a143-bdqq8         True           6m39s
org-giantswarm  │   │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-gvprp      True           5m43s
org-giantswarm  │   ├─Machine/gstempl1-worker-84d565588xb6n95-tm2k7            True           6m39s
org-giantswarm  │   │ ├─KubeadmConfig/gstempl1-worker-49eeb741-9cttg           True           6m39s
org-giantswarm  │   │ │ └─Secret/gstempl1-worker-49eeb741-9cttg                -              5m43s
org-giantswarm  │   │ └─VSphereMachine/gstempl1-default-4418a143-tmqpz         True           6m39s
org-giantswarm  │   │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-tm2k7      True           5m43s
org-giantswarm  │   └─Machine/gstempl1-worker-84d565588xb6n95-vh9mj            True           6m40s
org-giantswarm  │     ├─KubeadmConfig/gstempl1-worker-49eeb741-trd8d           True           6m40s
org-giantswarm  │     │ └─Secret/gstempl1-worker-49eeb741-trd8d                -              5m43s
org-giantswarm  │     └─VSphereMachine/gstempl1-default-4418a143-9ttrk         True           6m40s
org-giantswarm  │       └─VSphereVM/gstempl1-worker-84d565588xb6n95-vh9mj      True           5m43s
org-giantswarm  ├─MachineHealthCheck/gstempl1                                  -              7m1s
org-giantswarm  ├─VSphereCluster/gstempl1                                      True           7m1s
org-giantswarm  │ ├─AdmissionReport/1117c96f-4fd2-4b91-adb4-b5f4d9337576       -              7m1s
org-giantswarm  │ ├─BackgroundScanReport/1117c96f-4fd2-4b91-adb4-b5f4d9337576  -              6m29s
org-giantswarm  │ ├─Secret/gstempl1-credentials                                -              7m1s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-845gn       True           86s
org-giantswarm  │ │ └─VSphereVM/gstempl1-2pqt2                                 True           85s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-f5tgr       True           3m46s
org-giantswarm  │ │ └─VSphereVM/gstempl1-gw4j5                                 True           3m46s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-jlrxz       True           6m36s
org-giantswarm  │ │ └─VSphereVM/gstempl1-ts49x                                 True           6m35s
org-giantswarm  │ ├─VSphereMachine/gstempl1-default-4418a143-9ttrk             True           6m40s
org-giantswarm  │ │ └─VSphereVM/gstempl1-worker-84d565588xb6n95-vh9mj          True           5m43s
org-giantswarm  │ ├─VSphereMachine/gstempl1-default-4418a143-bdqq8             True           6m39s
org-giantswarm  │ │ └─VSphereVM/gstempl1-worker-84d565588xb6n95-gvprp          True           5m43s
org-giantswarm  │ └─VSphereMachine/gstempl1-default-4418a143-tmqpz             True           6m39s
org-giantswarm  │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-tm2k7          True           5m43s
org-giantswarm  ├─VSphereMachineTemplate/gstempl1-control-plane-05c119fe       -              7m1s
org-giantswarm  └─VSphereMachineTemplate/gstempl1-default-4418a143             -              7m1s

λ k get ipaddress -A
NAMESPACE        NAME       ADDRESS         POOL NAME   POOL KIND
org-giantswarm   gstempl1   10.10.222.232   wc-cp-ips   GlobalInClusterIPPool

λ k get globalinclusterippools.ipam.cluster.x-k8s.io 
NAME        ADDRESSES                         TOTAL   FREE   USED
wc-cp-ips   ["10.10.222.232-10.10.222.239"]   8       7      1

Should this change be mentioned in the release notes?

CHANGELOG.md has been updated

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

vxav · 2023-10-23T11:40:53Z

I don't see ipPoolName or networkName, we probably need them?

Out of scope:
Also the template is common to worker and control plane.
This made me think that we never had a need to run different images. Do we even need to allow it in the cluster chart interface?

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

jkremser · 2023-10-23T16:03:16Z

networkName:

flagVSphereNetworkName             = "vsphere-network-name"

so --vsphere-network-name=foo should do.

ipPoolName: yep this one is missing, I can add it
edit: should be there (651bf8e)

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

cmd/template/cluster/flag.go

erkanerol · 2023-10-23T18:52:21Z

Looks Good To Me in general. We can add/change flags to improve user experience later. I compared how it is implemented and saw it is the same as other CAPI providers, which is really nice. Thank you!

Please don't merge without getting review from honeybadger :)

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

vxav · 2023-10-24T06:56:16Z

Very nice indeed, this will be very useful !

uvegla · 2023-10-24T09:36:23Z

If not too much trouble, could you please add support for this in gitops add base command:

kubectl-gs/cmd/gitops/add/base/runner.go

Line 95 in 92d5379

    
           func generateClusterBaseTemplates(config common.StructureConfig) (common.ClusterBaseTemplates, error) {

I don't know how often gitops add base is used, it is just that it would be nice to have.

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

jkremser · 2023-10-24T12:02:16Z

@uvegla done in 1764739

uvegla · 2023-10-24T14:51:13Z

@jkremser Thank you! ❤️

vxav · 2023-10-25T08:25:55Z

cmd/template/cluster/flag.go

+	// VSphere only
+	cmd.Flags().StringVar(&f.VSphere.ControlPlane.IP, flagVSphereControlPlaneIP, "", "Control plane IP, leave empty for auto allocation.")
+	cmd.Flags().StringVar(&f.VSphere.ServiceLoadBalancerCIDR, flagVSphereServiceLoadBalancerCIDR, "", "CIDR for Service LB for new cluster")
+	cmd.Flags().StringVar(&f.VSphere.NetworkName, flagVSphereNetworkName, "grasshopper-capv", "Network name in vcenter that should be used for the new VMs")


I didn't pick it up but we should probably remove grasshopper-capv as a default value 🙂

jkremser requested a review from a team as a code owner October 13, 2023 10:13

jkremser force-pushed the update-vsphere branch 3 times, most recently from e49b640 to adb5f6a Compare October 13, 2023 10:30

jkremser requested a review from a team October 23, 2023 09:19

jkremser added 2 commits October 23, 2023 13:23

cluster template for vsphere

976bef9

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

CVE-2023-39325 > nancy-ignore

367df1d

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

jkremser force-pushed the update-vsphere branch 2 times, most recently from 1d4150b to c0be633 Compare October 23, 2023 11:31

add unit test

7a3669a

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

jkremser force-pushed the update-vsphere branch from c0be633 to 7a3669a Compare October 23, 2023 13:35

capv: add configurable ip pool name

651bf8e

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

erkanerol reviewed Oct 23, 2023

View reviewed changes

cmd/template/cluster/flag.go Outdated Show resolved Hide resolved

erkanerol approved these changes Oct 23, 2023

View reviewed changes

capv: Better logic for default k8s version

92d5379

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

vxav approved these changes Oct 24, 2023

View reviewed changes

uvegla approved these changes Oct 24, 2023

View reviewed changes

capv: add support also for 'k gs gitops add base --provider vsphere'

1764739

Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>

jkremser merged commit 1d04f4b into main Oct 24, 2023
1 check passed

jkremser deleted the update-vsphere branch October 24, 2023 12:21

jkremser mentioned this pull request Oct 24, 2023

Support kubectl gs on CAPV giantswarm/roadmap#2191

Closed

vxav reviewed Oct 25, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster template for vsphere #1129

cluster template for vsphere #1129

jkremser commented Oct 13, 2023 •

edited

Loading

vxav commented Oct 23, 2023

jkremser commented Oct 23, 2023 •

edited

Loading

erkanerol commented Oct 23, 2023 •

edited

Loading

vxav commented Oct 24, 2023

uvegla commented Oct 24, 2023 •

edited

Loading

jkremser commented Oct 24, 2023

uvegla commented Oct 24, 2023

vxav Oct 25, 2023

cluster template for vsphere #1129

cluster template for vsphere #1129

Conversation

jkremser commented Oct 13, 2023 • edited Loading

What does this PR do?

Should this change be mentioned in the release notes?

vxav commented Oct 23, 2023

jkremser commented Oct 23, 2023 • edited Loading

erkanerol commented Oct 23, 2023 • edited Loading

vxav commented Oct 24, 2023

uvegla commented Oct 24, 2023 • edited Loading

jkremser commented Oct 24, 2023

uvegla commented Oct 24, 2023

vxav Oct 25, 2023

Choose a reason for hiding this comment

jkremser commented Oct 13, 2023 •

edited

Loading

jkremser commented Oct 23, 2023 •

edited

Loading

erkanerol commented Oct 23, 2023 •

edited

Loading

uvegla commented Oct 24, 2023 •

edited

Loading