[sdk] can't directly use image as a parameter in ContainerSpec (need to cast to string)

[glemarivero]

Environment

kfp 2.7.0
kfp-pipeline-spec 0.3.0
kfp-server-api 2.0.5

Steps to reproduce

from kfp import compiler, dsl                                                                                                                                                                               
                                                                                                                                                                                                            
                                                                                                                                                                                                            
@dsl.container_component                                                                                                                                                                                    
def say_hello(image_uri: str):                                                                                                                                                                              
    """Log a greeting and return it as an output."""                                                                                                                                                        
                                                                                                                                                                                                            
    return dsl.ContainerSpec(                                                                                                                                                                               
        image=image_uri,                                                                                                                                                                                    
        command=['echo'], args=['Hello']                                                                                                                                                                    
    )                                                                                                                                                                                                       
                                                                                                                                                                                                            
                                                                                                                                                                                                            
@dsl.pipeline                                                                                                                                                                                               
def hello_pipeline(image_uri: str):                                                                                                                                                                         
    # greeting argument is provided automatically at runtime!                                                                                                                                               
    say_hello(image_uri=image_uri)                                                                                                                                                                          
                                                                                                                                                                                                            
compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')  

Error:

Traceback (most recent call last):                                                                                                                                                                          
  File "/home/glima/pipeline.py", line 15, in <module>                                                                                                                                                      
    def hello_pipeline(image_uri: str):                                                                                                                                                                     
  File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/pipeline_context.py", line 65, in pipeline                                                                                         return component_factory.create_graph_component_from_func(                                                                                                                                              
  File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/component_factory.py", line 673, in create_graph_component_from_func                                                           
    return graph_component.GraphComponent(                                                                                                                                                                    File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/graph_component.py", line 68, in __init__                                                                                      
    pipeline_spec, platform_spec = builder.create_pipeline_spec(                                                                                                                                            
  File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 1919, in create_pipeline_spec                                                                 build_spec_by_group(                                                                                                                                                                                    
  File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 1292, in build_spec_by_group                                                              
    subgroup_container_spec = build_container_spec_for_task(                                                                                                                                                  File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 608, in build_container_spec_for_task                                                     
    pipeline_spec_pb2.PipelineDeploymentConfig.PipelineContainerSpec(                                                                                                                                       
TypeError: bad argument type for built-in operation   

Expected result

Pipeline compiles correctly

Materials and Reference

If I cast the image argument of ContainerSpec then it works

from kfp import compiler, dsl                                                                                                                                                                               
                                                                                                                                                                                                            
                                                                                                                                                                                                            
@dsl.container_component                                                                                                                                                                                    
def say_hello(image_uri: str):                                                                                                                                                                              
    """Log a greeting and return it as an output."""                                                                                                                                                        
                                                                                                                                                                                                            
    return dsl.ContainerSpec(                                                                                                                                                                               
        image=str(image_uri),                                                                                                                                                                                    
        command=['echo'], args=['Hello']                                                                                                                                                                    
    )                                                                                                                                                                                                       
                                                                                                                                                                                                            
                                                                                                                                                                                                            
@dsl.pipeline                                                                                                                                                                                               
def hello_pipeline(image_uri: str):                                                                                                                                                                         
    # greeting argument is provided automatically at runtime!                                                                                                                                               
    say_hello(image_uri=image_uri)                                                                                                                                                                          
                                                                                                                                                                                                            
compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')  

Impacted by this bug? Give it a 👍.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[hahahannes]

Experiencing the same issue

[hahahannes]

I was trying to find the corresponding code (PipelineContainerSpec) but could not find it unfortunately

[hahahannes]

To me this seems to be a problem with how kfp compiles the pipeline.
So I tried to workaround this with

def foo(image: str):
    image2 = f"{image}"
    return dsl.ContainerSpec(image=image2)

which made the pipeline compile. But the pipeline did not run. I think it is not possible to use input parameters for the image.
The yaml looks like this:

    exec-foo:
      container:
        command:
        - sh
        image: '{{$.inputs.parameters[''image'']}}'

[hahahannes]

Seems to be related to #4433

[glemarivero]

but casting it as string like I did didn't work?

@dsl.container_component                                                                                                                                                                                    
def say_hello(image_uri: str):                                                                                                                                                                              
    """Log a greeting and return it as an output."""                                                                                                                                                        
                                                                                                                                                                                                            
    return dsl.ContainerSpec(                                                                                                                                                                               
        image=str(image_uri),                                                                                                                                                                                    
        command=['echo'], args=['Hello']                                                                                                                                                                    
    ) 

[hahahannes]

Hi @glemarivero Thanks for your help! It compiles but the container wont start. If I hardcode the image, then it works. In the other issue someone wrote Only command and args support placeholders

[glemarivero]

mmm strange, we've been using this in production with no issues..are you using these versions?

kfp 2.7.0
kfp-pipeline-spec 0.3.0
kfp-server-api 2.0.5

How does the pipeline look like in the UI? And what values do you get when you go into the job?
In my case I get gcr.io/ml-pipeline/google-clou.. in the UI, but when I click on the job, under Container Location (Worker pool 0 (chief)) I get the one I passed as a parameter

[hahahannes]

Interesting. I am running

kfp 2.9.0
kfp-pipeline-spec 0.4.0
kfp-server-api 2.0.5

In the input parameter section I get the correct image that works when I use it hardcoded. But I dont get any logs. So I assumed that parametrized images for container components does not work

[hahahannes]

I am alos getting logs from kiverno:

 Error node all-bwnmh.root.user-workload.executor: task 'all-bwnmh.root.user-workload.executor' errored: admission webhook "mutate.kyverno.svc-fail" denied the request: failed to add image information to the policy rule context: invalid image '{{$.inputs.parameters['image']}}' (bad image: docker.io/{{$.inputs.parameters['image']}}, defaultRegistry: docker.io, enableDefaultRegistryMutation: true: invalid reference format)

Which seems that the placeholder for the image did not get replaced

[glemarivero]

not sure if it's relevant but I'm using this in Vertex AI Pipelines, so maybe they've fixed it in their end

[hahahannes]

I see, maybe thats the reason, as I am running a self hosted kubeflow. Thank you nonetheless!

[hahahannes]

@HumairAK I am happy to contribute, would just need a pointer to the code. I guess this has to do with the templating of the pipeline spec?

[HumairAK]

@hahahannes apologies for the late reply, just saw this

If you're still interested in this, the offending code looks like it's here (when it's not casted). I think container specs are handled a bit differently than regular components in how parameters are resolved.

The issue is similar to #11404, except here before it gets to convert_to_placeholder(task.container_spec.image) it has already been converted to an input value type, and the convert_to_placeholder doesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.

Anyways that should give you a lead on how to handle the compilation side. As far as how this is handled in the backend, once a pipeline is submitted this still needs to be resolved. This will largely depend on how inputs are resolved for container types, if it's no different than components than the solution will likely utilize the helpers introduced as part of #11404 and make life easy for us here. You can confirm this by looking at the driver code changes in #11404 and debug this code once you have a pipeline yaml ready to submit with the string in the format {{$.inputs.parameters[''image'']}}.

Hopefully that gives you or the future assignee some leads, feel free to reach out here or slack (I'm a bit more responsive there).

[hahahannes]

Thank you very much @HumairAK. I have a look!

[hahahannes]

The issue is similar to #11404, except here before it gets to convert_to_placeholder(task.container_spec.image) it has already been converted to an input value type, and the convert_to_placeholder doesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.

This is correct! I was able to fix by checking the type. Now the client side compilation works! @HumairAK

[hahahannes]

I was able to setup a development environment for the kubeflow pipelines api-server and I can upload pipeline and create runs. I hoped to reach the point where the container image get set but unfortunately, it is not clear yet how it is related to the driver (backend/srv/v2/driver). I see that the Argo Workflow gets prepared and I expect the image to be set there.

[zeidsolh]

The issue is similar to #11404, except here before it gets to convert_to_placeholder(task.container_spec.image) it has already been converted to an input value type, and the convert_to_placeholder doesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.

This is correct! I was able to fix by checking the type. Now the client side compilation works! @HumairAK

Would you mind telling me how to do that to test it out on my side? Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.