Use EKS Addon to manage kube-proxy and coredns

[flostadler]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Right now kube-proxy and coredns are self-managed. This has the drawback that you cannot keep those cluster components updated via pulumi.

Additionally, this requires us to do a complex process of shelling out to kubectl to patch the coredns deployment. This is required so coredns can run on fargate:

pulumi-eks/nodejs/eks/cluster.ts

Lines 967 to 1024 in 986c528

	pulumi.all([result.id, selectors, kubeconfig]).apply(([_, sels, kconfig]) => {
	// Only patch CoreDNS if there is a selector in the FargateProfile which causes
	// `kube-system` pods to launch in Fargate.
	if (sels.findIndex((s) => s.namespace === "kube-system") !== -1) {
	// Only do the imperative patching during deployments, not previews.
	if (!pulumi.runtime.isDryRun()) {
	// Write the kubeconfig to a tmp file and use it to patch the `coredns`
	// deployment that AWS deployed already as part of cluster creation.
	const tmpKubeconfig = tmp.fileSync();
	fs.writeFileSync(tmpKubeconfig.fd, JSON.stringify(kconfig));
	// Determine if the CoreDNS deployment has a computeType annotation.
	const cmdGetAnnos = `kubectl get deployment coredns -n kube-system -o jsonpath='{.spec.template.metadata.annotations}'`;
	const getAnnosOutput = childProcess.execSync(cmdGetAnnos, {
	env: {
	...process.env,
	KUBECONFIG: tmpKubeconfig.name,
	},
	});
	const getAnnosOutputStr = getAnnosOutput.toString();
	// See if getAnnosOutputStr contains the annotation we're looking for.
	if (!getAnnosOutputStr.includes("eks.amazonaws.com/compute-type")) {
	// No need to patch the deployment object since the annotation is not present. However, we need to re-create the CoreDNS pods since
	// the existing pods were created before the FargateProfile was created, and therefore will not have been scheduled by fargate-scheduler.
	// See: https://github.com/pulumi/pulumi-eks/issues/1030.
	const cmd = `kubectl rollout restart deployment coredns -n kube-system`;
	childProcess.execSync(cmd, {
	env: {
	...process.env,
	KUBECONFIG: tmpKubeconfig.name,
	},
	});
	return;
	}
	const patch = [
	{
	op: "remove",
	path: "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type",
	},
	];
	const cmd = `kubectl patch deployment coredns -n kube-system --type json -p='${JSON.stringify(
	patch,
	)}'`;
	childProcess.execSync(cmd, {
	env: {
	...process.env,
	KUBECONFIG: tmpKubeconfig.name,
	},
	});
	}
	}
	});
	}
	return result;
	});

Instead of doing that, we can manage those components as EKS addons and configure them with the pulumi resource aws.eks.Addon.

E.g. for enabling coredns to run on Fargate we should set:

const example = new aws.eks.Addon("example", {
    clusterName: "mycluster",
    addonName: "coredns",
    ...
    configurationValues: JSON.stringify({
        computeType: "Fargate"
    })
};

Affected area/feature

[flostadler]

This will also solve #1294

[corymhall]

merged into release branch