Use of cert manager instead of generating CA bundle during build

[anekdoti]

The documentation for webhooks advises to just add the generated files ca.pem and ca-key.pem to the repository of your operator, and (implicitly) to add them to your GitOps repository together with the kustomize script that generates a secret out of them. I understand that it is no security in this case, as the CA is only used cluster-internally, but it is still somehow an anti pattern.

In our cluster infrastructure, we have the cert manager available. Do you have any hints how to generate the CA + key using cert manager using the custom resource "Certificate" of cert manager? Did somebody already try this and maybe even have a sample resource? What are things to keep an eye on?

[buehler]

Hmm. I see the point. I just learned that Kubernetes has its own PKI, but that does not automatically approve CSRs.

So theoretically you could overwrite the secret that is generated with the CA cert/key combination. I don't know if it is easily possible, but you could try to patch the deployment resource in a way to overwrite the required secret that mounts the certificate. That way, you can generate the certificate with the cert manager and then mount it into the operator.

But this is a good idea that we create a configuration for this. I assume there are others that would use "real" certificates.

[erin-allison]

That seems like a documentation error to me. If you use the auto-generated operator definition, there is an init container which pulls down cfssl and generates a set of certificates for the webhook to use.

dotnet-operator-sdk/src/KubeOps/Operator/Commands/Generators/OperatorGenerator.cs

Lines 119 to 153 in 02bb7ec

	InitContainers = !_hasWebhooks
	? null
	: new List<V1Container>
	{
	new()
	{
	Image = "operator",
	Name = "webhook-installer",
	Args = new[]
	{
	"webhooks",
	"install",
	},
	Env = new List<V1EnvVar>
	{
	new()
	{
	Name = "POD_NAMESPACE",
	ValueFrom = new V1EnvVarSource
	{
	FieldRef = new V1ObjectFieldSelector
	{
	FieldPath = "metadata.namespace",
	},
	},
	},
	},
	VolumeMounts = new List<V1VolumeMount>
	{
	new()
	{
	Name = "certificates",
	MountPath = "/certs",
	},
	new()

dotnet-operator-sdk/src/KubeOps/Operator/Commands/Management/Webhooks/Install.cs

Lines 54 to 72 in 02bb7ec

	public async Task<int> OnExecuteAsync(CommandLineApplication app)
	{
	var @namespace = await _client.GetCurrentNamespace();
	using var certManager = new CertificateGenerator(app.Out);
	#if DEBUG
	CertificatesPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
	CaCertificatesPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
	await certManager.CreateCaCertificateAsync(CaCertificatesPath);
	#endif
	Directory.CreateDirectory(CertificatesPath);
	File.Copy(Path.Join(CaCertificatesPath, "ca.pem"), Path.Join(Path.Join(CertificatesPath, "ca.pem")));
	await certManager.CreateServerCertificateAsync(
	CertificatesPath,
	_settings.Name,
	@namespace,
	Path.Join(CaCertificatesPath, "ca.pem"),
	Path.Join(CaCertificatesPath, "ca-key.pem"));

dotnet-operator-sdk/src/KubeOps/Operator/Commands/CommandHelpers/CertificateGenerator.cs

Lines 93 to 128 in 02bb7ec

	public async Task CreateServerCertificateAsync(
	string outputFolder,
	string name,
	string @namespace,
	string caPath,
	string caKeyPath)
	{
	if (!_initialized)
	{
	await PrepareExecutables();
	_initialized = true;
	}
	Directory.CreateDirectory(outputFolder);
	_servercsr = Path.Join(_tempDirectory, Path.GetRandomFileName());
	await using (var serverCsrStream = new StreamWriter(new FileStream(_servercsr, FileMode.CreateNew)))
	{
	await serverCsrStream.WriteLineAsync(
	ServerCsr($"{name}.{@namespace}.svc", $"*.{@namespace}.svc", "*.svc"));
	}
	await _appOut.WriteLineAsync(
	$@"Generating server certificate for ""{name}"" in namespace ""{@namespace}"".");
	await ExecuteProcess(
	new Process
	{
	StartInfo = new ProcessStartInfo
	{
	WorkingDirectory = outputFolder,
	FileName = ShellExecutor,
	Arguments = RuntimeInformation.IsOSPlatform(OSPlatform.Windows)
	? $"/c {_cfssl} gencert -ca=file:{caPath.Replace('\\', '/')} -ca-key=file:{caKeyPath.Replace('\\', '/')} -config={_caconfig} -profile=server {_servercsr} | {_cfssljson} -bare server -"
	: $@"-c ""{_cfssl} gencert -ca={caPath} -ca-key={caKeyPath} -config={_caconfig} -profile=server {_servercsr} | {_cfssljson} -bare server -""",
	},
	});

[erin-allison]

You could use cert-manager if you wanted to customize the deployment files; you would simply need to remove the init container and mount a secret as a folder (or files) in the right spot on the operator pod(s).

EDIT: I realized after a bit more reading that actually wouldn't work as well. The init container does more than just certificates; it is responsible for creating and applying the webhook definitions.

[erin-allison]

@buehler I do agree with allowing certificate generation to be disabled, but would like to see that accompanied with details of where exactly the external certificate files would need to be mounted (and what they would need to be named) to have the webhook still function.

[buehler]

That is a fine idea. If we just disable the generation of the certs, a user can create and mount them on their own. Let me think about this.

[erin-allison]

The best way to do that would probably be an additional argument for the webhook install function, as the decision on whether or not to generate certificates is more at run-time, not design-time. From there, someone could just tack the argument on in the generated deployment.yaml