[BUG] Default pod template init containers no longer work in Flyte-binary 1.14.1

[nuhaj]

Describe the bug

Upgrading from Flyte Binary 1.13 to 1.14 broke our default pod template where initContainers was being initialized at runtime. Using compile time pod specs is an option but that would involve pushing the responsibility of pod templating to our flyte users.

This issue explains a bit of how we got init containers to work although there was no documented support in 1.13
#5376 (comment)

This PR adding default-init and primary-init support to compile time specs seems to be the cause https://github.com/flyteorg/flyte/pull/5750/files

sample pod template

apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-workflow-base
  namespace: flyte
template:
  metadata:
    name: flyte-workflow-base
  spec:
    initContainers:
      - name: init
        image: alpine
        volumeMounts:
        - name: shared-data
          mountPath: /data
    containers:
      - name: default 
        image: rwgrim/docker-noop     
        volumeMounts:
          - name: shared-data
            mountPath: /data
        terminationMessagePath: "/dev/foo"
    ....

Expected behavior

I would expect default default pod templates to still be an option according to https://docs.flyte.org/en/latest/deployment/configuration/general.html#evaluation-order-in-podtemplates
I suspect the newer changes will always take precedence over pod templates

Additional context to reproduce

To reproduce create a default pod template with init container. Set in plugins.k8 and clusterResourceTemplates.inline confirm initcontainer using describe pod. Upgrade to 1.14 using the same helm and pod template. InitContainer will be ignored and you will get an Access Denied error attempting to write to the volume mount defined in it.
Further discussion https://flyte-org.slack.com/archives/CP2HDHKE1/p1738100120966119

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Sovietaced]

Per the docs, containers or init containers other than default and primary will be ignored: https://docs.flyte.org/en/latest/deployment/configuration/general.html#create-a-podtemplate-resource

The containers field is required in each k8s PodSpec. If no default configuration is desired, specifying a container with a name other than “default” or “primary” (for example, “noop”) is considered best practice. Since Flyte only processes the “default” or “primary” containers, this value will always be dropped during Pod construction.

So if we want to support injecting non-flyte containers using pod templates that would be new behavior.