Skip to content

Commit

Permalink
feat(docs): Add architecture diagram (#11490)
Browse files Browse the repository at this point in the history
* Fix minor lint errors

Signed-off-by: Anish Asthana <anishasthana1@gmail.com>

* Add Kubeflow Pipelines Architecture

Signed-off-by: Anish Asthana <anishasthana1@gmail.com>

* Update community section

Signed-off-by: Anish Asthana <anishasthana1@gmail.com>

---------

Signed-off-by: Anish Asthana <anishasthana1@gmail.com>
  • Loading branch information
anishasthana authored Jan 17, 2025
1 parent a0e2406 commit 3e423d8
Show file tree
Hide file tree
Showing 5 changed files with 303 additions and 8 deletions.
26 changes: 18 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Kubeflow Pipelines

[![Coverage Status](https://coveralls.io/repos/github/kubeflow/pipelines/badge.svg?branch=master)](https://coveralls.io/github/kubeflow/pipelines?branch=master)
[![SDK Documentation Status](https://readthedocs.org/projects/kubeflow-pipelines/badge/?version=latest)](https://kubeflow-pipelines.readthedocs.io/en/stable/?badge=latest)
[![SDK Package version](https://img.shields.io/pypi/v/kfp?color=%2334D058&label=pypi%20package)](https://pypi.org/project/kfp)
Expand All @@ -19,8 +21,7 @@ The Kubeflow pipelines service has the following goals:

* Kubeflow Pipelines can be installed as part of the [Kubeflow Platform](https://www.kubeflow.org/docs/started/installing-kubeflow/#kubeflow-platform). Alternatively you can deploy [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/) as a standalone service.


* The Docker container runtime has been deprecated on Kubernetes 1.20+. Kubeflow Pipelines has switched to use [Emissary Executor](https://www.kubeflow.org/docs/components/pipelines/legacy-v1/installation/choose-executor/#emissary-executor) by default from Kubeflow Pipelines 1.8. Emissary executor is Container runtime agnostic, meaning you are able to run Kubeflow Pipelines on Kubernetes cluster with any [Container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/).
* The Docker container runtime has been deprecated on Kubernetes 1.20+. Kubeflow Pipelines has switched to use [Emissary Executor](https://www.kubeflow.org/docs/components/pipelines/legacy-v1/installation/choose-executor/#emissary-executor) by default from Kubeflow Pipelines 1.8. Emissary executor is Container runtime agnostic, meaning you are able to run Kubeflow Pipelines on Kubernetes cluster with any [Container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/).

## Documentation

Expand All @@ -34,19 +35,27 @@ Consult the [Python SDK reference docs](https://kubeflow-pipelines.readthedocs.i

## Contributing to Kubeflow Pipelines

Before you start contributing to Kubeflow Pipelines, read the guidelines in [How to Contribute](./CONTRIBUTING.md). To learn how to build and deploy Kubeflow Pipelines from source code, read the [developer guide](./developer_guide.md).
Before you start contributing to Kubeflow Pipelines, read the guidelines in [How to Contribute](./CONTRIBUTING.md). To learn how to build and deploy Kubeflow Pipelines from source code, reagd the [developer guide](./developer_guide.md).

## Kubeflow Pipelines Community

### Community Meeting

The Kubeflow Pipelines Community Meeting occurs every other Wed 10-11AM (PST).

## Kubeflow Pipelines Community Meeting
[Calendar Invite](https://calendar.google.com/event?action=TEMPLATE&tmeid=NTdoNG5uMDBtcnJlYmdlOWt1c2lkY25jdmlfMjAxOTExMTNUMTgwMDAwWiBqZXNzaWV6aHVAZ29vZ2xlLmNvbQ&tmsrc=jessiezhu%40google.com&scp=ALL)

The meeting is happening every other Wed 10-11AM (PST)
[Calendar Invite](https://calendar.google.com/event?action=TEMPLATE&tmeid=NTdoNG5uMDBtcnJlYmdlOWt1c2lkY25jdmlfMjAxOTExMTNUMTgwMDAwWiBqZXNzaWV6aHVAZ29vZ2xlLmNvbQ&tmsrc=jessiezhu%40google.com&scp=ALL) or [Join Meeting Directly](https://meet.google.com/phd-ixfj-kcr/)
[Direct Meeting Link](https://zoom.us/j/92607298595?pwd%3DVlKLUbiguGkbT9oKbaoDmCxrhbRop7.1&sa=D&source=calendar&ust=1736264977415448&usg=AOvVaw1EIkjFsKy0d4yQPptIJS3x)

[Meeting notes](http://bit.ly/kfp-meeting-notes)

## Kubeflow Pipelines Slack Channel
### Slack

[#kubeflow-pipelines](https://kubeflow.slack.com)
We also have a slack channel (#kubeflow-pipelines) on the Cloud Native Computing Foundation Slack workspace. You can find more details at [https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels](https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels)

## Architecture

Details about the KFP Architecture can be found at [Architecture.md](docs/Architecture.md)

## Blog posts

Expand All @@ -56,6 +65,7 @@ The meeting is happening every other Wed 10-11AM (PST)
* [Part 2: How to deploy Jupyter notebooks as components of a Kubeflow ML pipeline](https://towardsdatascience.com/how-to-deploy-jupyter-notebooks-as-components-of-a-kubeflow-ml-pipeline-part-2-b1df77f4e5b3)
* [Part 3: How to carry out CI/CD in Machine Learning (“MLOps”) using Kubeflow ML pipelines](https://medium.com/google-cloud/how-to-carry-out-ci-cd-in-machine-learning-mlops-using-kubeflow-ml-pipelines-part-3-bdaf68082112)
* [Tekton optimizations for Kubeflow Pipelines 2.0](https://developer.ibm.com/blogs/awb-tekton-optimizations-for-kubeflow-pipelines-2-0) (By Tommy Li)

## Acknowledgments

Kubeflow pipelines uses [Argo Workflows](https://github.com/argoproj/argo-workflows) by default under the hood to orchestrate Kubernetes resources. The Argo community has been very supportive and we are very grateful. Additionally there is Tekton backend available as well. To access it, please refer to [Kubeflow Pipelines with Tekton repository](https://github.com/kubeflow/kfp-tekton).
7 changes: 7 additions & 0 deletions docs/Architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Kubeflow Pipelines Architecture

Below is a detailed diagram of the Kubeflow Pipelines Architecture.

![KubeFlow Pipelines Cluster Wide Architecture](../images/kfp-cluster-wide-architecture.png)

The above diagram documents the various components in Kubeflow Pipelines and how they interact. This should be a useful starting point for any developers.
8 changes: 8 additions & 0 deletions images/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# KFP Architecture Diagram

To make edits to the KFP architecture, do the following:

1. Import kfp-cluster-wide-architecture.drawio.xml into your drawio environment.
2. Make any required updates to the architecture
3. Export a new png as well as xml of the diagram
4. Overwrite the existing files in this directory
Loading

0 comments on commit 3e423d8

Please sign in to comment.