feat: recreate agent pods if agent image is updated

[starbops]

IMPORTANT: Please do not create a Pull Request without creating an issue first.

Problem:

Currently, upgrading the vm-dhcp-controller does not upgrade the agents simultaneously, resulting in inconsistent versions between the controller and the agent.

Solution:

The IPPool object will contain additional fields holding the information of the agent Pod. The controller will also propagate the image name changes to all IPPool objects and recreate the agent Pods according to the up-to-date image name to ensure the running agents match the agent Pod reference in their corresponding IPPool objects.

Related Issue:

harvester/harvester#5058

Test plan:

$ kubectl -n harvester-system get deploy harvester-vm-dhcp-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
rancher/harvester-vm-dhcp-controller:main-head

$ kubectl -n harvester-system get pods default-net-48-agent -o jsonpath='{.spec.containers[0].image}'  
rancher/harvester-vm-dhcp-agent:main-head

Upgrade the vm-dhcp-controller chart:

helm upgrade --install harvester-vm-dhcp-controller ./chart -f values.yaml --namespace=harvester-system --create-namespace

After the upgrade, check if the agent Pods are using the newer image:

$ kubectl -n harvester-system get deploy harvester-vm-dhcp-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
starbops/harvester-vm-dhcp-controller:agent-upgrade-head

$ kubectl -n harvester-system get pods default-net-48-agent -o jsonpath='{.spec.containers[0].image}'                       
starbops/harvester-vm-dhcp-agent:agent-upgrade-head

[bk201]

Maybe compare the container image with the agent image name stored in the controller?

Suggested change

	if agentPod.GetUID() != ipPool.Status.AgentPodRef.UID || agentPod.Spec.Containers[0].Image != ipPool.Status.AgentPodRef.Image {
	if agentPod.GetUID() != ipPool.Status.AgentPodRef.UID || agentPod.Spec.Containers[0].Image != h.agentImage.String() {

How I tested

I check out the PR with the test1 branch, build and publish to my repo, and edit the values.file:

image:
  repository: bk201z/harvester-vm-dhcp-controller
  pullPolicy: Always
  # Overrides the image tag whose default is the chart appVersion.
  tag: "test1-head"

agent:
  image:
    repository: bk201z/harvester-vm-dhcp-agent
    pullPolicy: Always
    tag: "test1-head"

webhook:
  replicaCount: 1
  image:
    repository: bk201z/harvester-vm-dhcp-webhook
    pullPolicy: Always
    tag: "test1-head"

Install chart and create an IPPool

helm upgrade --install harvester-vm-dhcp-controller ./chart --namespace=harvester-system --create-namespace

Check out branch test2 and re-publish, and edit the values file again:

image:
  repository: bk201z/harvester-vm-dhcp-controller
  pullPolicy: Always
  # Overrides the image tag whose default is the chart appVersion.
  tag: "test2-head"

agent:
  image:
    repository: bk201z/harvester-vm-dhcp-agent
    pullPolicy: Always
    tag: "test2-head"

webhook:
  replicaCount: 1
  image:
    repository: bk201z/harvester-vm-dhcp-webhook
    pullPolicy: Always
    tag: "test2-head"

And then upgrade the chart:

helm upgrade --install harvester-vm-dhcp-controller ./chart --namespace=harvester-system --create-namespace

The agent pod seems not to be restarted.

[starbops]

Since the agent image information is an internal controller variable (not in an object's spec), we need to reflect it on the object status first (done by the ippool-register reconcile loop). Then, rely on the other reconciliation loop to sync the status with the real pod.

It's the ippool-agent-monitor reconcile loop's mission to ensure that the running agent pod does not drift away. If it finds the agent pod it tracks does not match what's recorded in the AgentPodRef, it deletes the pod.

Thank you for pointing out there's a bug that the AgentPodRef is not updated if the agent pod already exists. I've updated the relevant part. Please take a look :)

[Yu-Jack]

LGTM, thanks!

[bk201]

lgtm!