Teleport during MC creation, replace need for VPN

[gawertm]

Gathering info from Erkan.

Summary:

• During private MC creation in CAPZ, VPN is needed to access the new MCs API server for mc-bootstrap to complete the installation.
• Later, VPN can be removed and teleport is enabled. However, VPN is needed for mc-bootstrap to finish the installation.

Blocker

• VPN is again needed for private MC creation.

Action Items

• Figure out a way to get access to private MC during the mc-bootstrap creation process.

[gawertm]

@tuladhar I saw your PR https://github.com/giantswarm/mc-bootstrap/pull/805
this is only for a private WC, right?

the goal of this task is to bootstrap a MC without prior VPN, right? Just to make sure we understood the same thing :)

[tuladhar]

@tuladhar I saw your PR https://github.com/giantswarm/mc-bootstrap/pull/805
this is only for a private WC, right?

the goal of this task is to bootstrap a MC without prior VPN, right? Just to make sure we understood the same thing :)

Yes, that's correct.

[stone-z]

Spyros plans to pick this up again after the teleport upgrade

[stone-z]

With the v16 upgrade and Spyros's work to allow SSH before the cluster starts, this is now theoretically possible. A VM containing our flatcar image could be started inside a customer subnet and joined to our teleport cluster using the same mechanism as we use prior to cluster startup. The customer would just need the VM image and a join token.

The last piece missing is the delivery of the "teleport box" to a customer. Spyros would like to sync with Team Rocket to understand what an optimal solution might look like, based on customer environments and past experience.

[gawertm]

I know it was initially my idea, but from recent customers we learned that they might be a bit careful allowing us to just install a backdoor into their environment.. also the current customers are ok with the VPN and we do not have a new one in the pipeline anymore. My opinion is that we should keep it in the backlog and revisit this idea if we have a customer who would not allow VPN but would allow this solution.

@giantswarm/team-rocket What do you think?

[vxav]

We could probe our existing on-prem customers and see how acceptable a concept it would be for their security team.
That would give us a sense of whether it's worth investing time in it or if we'll get shot down every time?

[stone-z]

I'd be interested to hear from the on-prem customers if this makes any difference. IIUC, this is mostly about manageability for us, right? Functionally, is there any difference between a teleport host and the existing VPN bastions? We need some way to reach the nodes and the underlying virtualization consoles and teleport is effectively just a different client to do the same thing. Or do we not use bastions in newer on-prem setups?

That said, getting rid of the VPN would be nice but it isn't urgent for me if Rocket is comfortable maintaining it. It hasn't needed that much effort over the years, and I'm not really "worried" about it, other than the fact that not many people know how to work on it anymore and it has a [low likelihood] + [very high impact] kind of risk profile. Replacing it wouldn't really change the risk, but would mean fewer niches for Giant Swarm to keep track of.

[stone-z]

@giantswarm/team-rocket I'm going to park this for now, give us a shout if it becomes relevant again.

To us, it seems like "teleport on a VM" is basically equivalent to "openvpn on a VM" from a customer security perspective, so in theory we can ditch bastions in favor of such a thing. But we wouldn't want to spend time on it if customers won't use it or it's not that important after all, or if Zach is just completely missing the point here 😄

[gawertm]

@glitchcrab will be able to say more, but we are not using bastion hosts like in the past anymore. With some clients we do have a site2site VPN setup. With others they provide a point2site vpn, terminating at like a citrix host or whatever, so like a bastion host but managed by the customer.

[glitchcrab]

@glitchcrab will be able to say more, but we are not using bastion hosts like in the past anymore. With some clients we do have a site2site VPN setup. With others they provide a point2site vpn, terminating at like a citrix host or whatever, so like a bastion host but managed by the customer.

It depends on the customer, but no our default is not to use any kind of bastion host.