Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): run pipeline from CLI directly #212

Merged
merged 1 commit into from
Dec 2, 2024

Conversation

tumido
Copy link
Member

@tumido tumido commented Nov 26, 2024

Follow up to: #202 (comment)

@MichaelClifford, @Shreyanand can you please help me understand the default values we set here and if ALL of them make sense in a generic environment. I've made a check-list in code via FIXME: comments. Be aware that the default values are not necessarily expected to work in the MOC cluster/ilab namespace context. (Removed from the PR, we can revisit this separatelly.)

This PR allows us to submit the pipeline run directly via CLI with no drag-and-drop-in-UI-slavery anymore...

All params are optional but allows you to control things:

$ python pipeline.py run --help
Usage: pipeline.py run [OPTIONS]

  Run the pipeline immediately against current kubernetes context (cluster and
  namespace).

  Command sets expected dev-cluster friendly default values when submitting.

Options:
  --mock [sdg|train|eval]  Mock part of the pipeline
  -e, --experiment TEXT    Set KFP experiment name.
  -r, --run TEXT           Set KFP run name.
  -p, --param TEXT         Override default parameters in KEY=VALUE format.
                           Default parameters are suitable for dev cluster -
                           the MOC cluster, `ilab` namespace.
  --help                   Show this message and exit.

Submitting a run then looks like this:

$ python pipeline.py run
Experiment details: https://ds-pipeline-dspa-ilab.apps.ocp-beta-test.nerc.mghpcc.org/#/experiments/details/f3fb689f-8f38-4822-9b6c-aa1c51d8d121
Run details: https://ds-pipeline-dspa-ilab.apps.ocp-beta-test.nerc.mghpcc.org/#/runs/details/2e5bb117-3a3d-44ed-bd11-49864f16df45

Copy link
Collaborator

@leseb leseb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is that different from using the kfp CLI?

utils/kfp_client.py Outdated Show resolved Hide resolved
pipeline.py Outdated Show resolved Hide resolved
pipeline.py Show resolved Hide resolved
@tumido
Copy link
Member Author

tumido commented Nov 26, 2024

@leseb

How is that different from using the kfp CLI?

CLI will become usable against OCP once kubeflow/pipelines#11400 get's into kfp release. Before that there's no way to authenticate KFP CLI properly. And since we're locked on 2.9 at the moment, we can't benefit from it any time soon.

Signed-off-by: Tomas Coufal <tcoufal@redhat.com>
@tumido tumido changed the title wip: feat(cli): run pipeline from CLI directly feat(cli): run pipeline from CLI directly Dec 2, 2024
@tumido tumido requested a review from leseb December 2, 2024 12:32
utils/kfp_client.py Show resolved Hide resolved
@tumido tumido merged commit 4c88800 into opendatahub-io:main Dec 2, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants