Skip to content

Commit

Permalink
Add hera tour to walkthrough (#1002)
Browse files Browse the repository at this point in the history
* hera-tour part of walkthrough based on upcoming ArgoCon EU 2024 talk,
link will be up after
* Add `use_cases/testing_templates_and_workflows.py` to show how to test
script templates and workflows
* Tidy up other examples which will used in the talk

---------

Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
  • Loading branch information
elliotgunton authored Mar 14, 2024
1 parent 5c252ac commit 61bf73c
Show file tree
Hide file tree
Showing 14 changed files with 622 additions and 30 deletions.
55 changes: 52 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ respected. To solve this, add the following as a config in your `.vscode/launch.
```json
{
"name": "Debug Tests",
"type": "python",
"type": "debugpy", // "python" is now deprecated
"request": "launch",
"purpose": ["debug-test"],
"console": "integratedTerminal",
Expand All @@ -69,7 +69,7 @@ Please keep in mind the following guidelines and practices when contributing to
1. Add unit tests for any new code you write.
1. Add an example, or extend an existing example, with any new features you may add. Use `make examples` to ensure that the documentation and examples are in sync.

## Adding new Workflow tests
## Adding new Workflow YAML generation tests

Hera has an automated-test harness that is coupled with our documentation. In order to add new tests, please follow these steps -

Expand All @@ -88,7 +88,10 @@ In order to add a new workflow test to test Hera functionality, do the following

### Upstream Hera examples

Tests that correspond to any [upstream Argo Workflow examples](https://github.com/argoproj/argo-workflows/tree/main/examples) should live in `examples/workflows/upstream/*.py`. These tests exist to ensure that Hera has complete parity with Argo Workflows and also to catch any regressions that might happen.
Tests that correspond to any
[upstream Argo Workflow examples](https://github.com/argoproj/argo-workflows/tree/main/examples) should live in
`examples/workflows/upstream/*.py`. These tests exist to ensure that Hera has complete parity with Argo Workflows and
also to catch any regressions that might happen.

In order to add a new workflow test to test Hera functionality, do the following -

Expand All @@ -104,6 +107,52 @@ In order to add a new workflow test to test Hera functionality, do the following
* If you would like to update the golden copy of the test files, you can run `make regenerate-test-data`
* The golden copies must be checked in to ensure that regressions may be caught in the future

## Adding new Workflow on-cluster tests

Hera's CICD spins up Argo Workflows on a local Kubernetes cluster, which runs tests decorated with
`@pytest.mark.on_cluster`. If you want to add more on-cluster tests, the easiest way is through a GitHub Codespace. You
can then run the same `make` commands that run in CICD:

```
make install-k3d
```

This will install the k3d CLI.

```
make run-argo
```

This will create a cluster using k3d called `test-cluster`, then create a namespace called `argo` on it, applying the
argo configuration, and patching the deployment to use `server` as the `auth-mode`, meaning the connection to submit the
workflow doesn't require an authentication mechanism.

You can then run existing on-cluster tests to ensure everything is set up correctly. This command also ports-forward the
server's port.

```
make test-on-cluster
```

### Viewing the Argo UI from a Codespace

> Before doing this, note that **anyone** will be able to connect using the Argo UI URL!
Ensure Argo Workflows is running using the `make` command:

```
make run-argo
```

Forward the Server's port using kubectl:

```
kubectl -n argo port-forward deployment/argo-server 2746:2746
```

Then, go to the `PORTS` panel in VSCode, and add the `2746` port. You should see a green circle to the left of the port.
Then right click on the `2746` row and set `Port Visibility` to `public`. You can then open the URL in your browser to view the Argo UI.

## Code of Conduct

Please be mindful of, and adhere to, the CNCF's
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ run-argo: ## Start the argo server
kubectl get namespace argo || kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v$(ARGO_WORKFLOWS_VERSION)/install.yaml
kubectl patch deployment argo-server --namespace argo --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["server", "--auth-mode=server"]}]'
kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default --namespace=argo
kubectl rollout status -n argo deployment/argo-server --timeout=120s --watch=true

.PHONY: stop-argo
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,10 @@
=== "Hera"

```python linenums="1"
from hera.workflows import (
DAG,
Workflow,
script,
)
from hera.workflows import DAG, Workflow, script


@script(add_cwd_to_sys_path=False, image="python:alpine3.6")
@script(image="python:3.12")
def echo(message):
print(message)

Expand Down Expand Up @@ -76,8 +72,11 @@
script:
command:
- python
image: python:alpine3.6
image: python:3.12
source: |-
import os
import sys
sys.path.append(os.getcwd())
import json
try: message = json.loads(r'''{{inputs.parameters.message}}''')
except: message = r'''{{inputs.parameters.message}}'''
Expand Down
5 changes: 1 addition & 4 deletions docs/examples/workflows/upstream/dag_diamond.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,7 @@ The upstream example can be [found here](https://github.com/argoproj/argo-workfl
```python linenums="1"
from hera.workflows import DAG, Container, Parameter, Workflow

with Workflow(
generate_name="dag-diamond-",
entrypoint="diamond",
) as w:
with Workflow(generate_name="dag-diamond-", entrypoint="diamond") as w:
echo = Container(
name="echo",
image="alpine:3.7",
Expand Down
111 changes: 111 additions & 0 deletions docs/examples/workflows/use-cases/testing_templates_and_workflows.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Testing Templates And Workflows






=== "Hera"

```python linenums="1"
from hera.shared import global_config
from hera.workflows import DAG, RunnerScriptConstructor, Script, Workflow, WorkflowsService, script

try:
from pydantic.v1 import BaseModel
except ImportError:
from pydantic import BaseModel

global_config.set_class_defaults(Script, constructor=RunnerScriptConstructor())


class Rectangle(BaseModel):
length: float
width: float

def area(self) -> float:
return self.length * self.width


@script(constructor="runner", image="my-built-python-image")
def calculate_area_of_rectangle(rectangle: Rectangle) -> float:
return rectangle.area()


with Workflow(
generate_name="dag-",
entrypoint="dag",
namespace="argo",
workflows_service=WorkflowsService(host="https://localhost:2746"),
) as w:
with DAG(name="dag"):
A = calculate_area_of_rectangle(
name="rectangle-1", arguments={"rectangle": Rectangle(length=1.2, width=3.4).json()}
)
B = calculate_area_of_rectangle(
name="rectangle-2", arguments={"rectangle": Rectangle(length=4.3, width=2.1).json()}
)
A >> B


def test_calculate_area_of_rectangle():
r = Rectangle(length=2.0, width=3.0)
assert calculate_area_of_rectangle(r) == 6.0


def test_create_workflow():
model_workflow = w.create(wait=True)
assert model_workflow.status and model_workflow.status.phase == "Succeeded"

echo_node = next(
filter(
lambda n: n.display_name == "echo",
model_workflow.status.nodes.values(),
)
)
assert echo_node.outputs.parameters[0].value == "my value"
```

=== "YAML"

```yaml linenums="1"
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-
namespace: argo
spec:
entrypoint: dag
templates:
- dag:
tasks:
- arguments:
parameters:
- name: rectangle
value: '{"length": 1.2, "width": 3.4}'
name: rectangle-1
template: calculate-area-of-rectangle
- arguments:
parameters:
- name: rectangle
value: '{"length": 4.3, "width": 2.1}'
depends: rectangle-1
name: rectangle-2
template: calculate-area-of-rectangle
name: dag
- inputs:
parameters:
- name: rectangle
name: calculate-area-of-rectangle
script:
args:
- -m
- hera.workflows.runner
- -e
- examples.workflows.use_cases.testing_templates_and_workflows:calculate_area_of_rectangle
command:
- python
image: my-built-python-image
source: '{{inputs.parameters}}'
```

2 changes: 2 additions & 0 deletions docs/walk-through/about-hera.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,5 @@ functionality in Hera. Read more in the [user guides](../user-guides/core-concep
A natural extension of a Python DSL for Argo is tighter integration with Python scripts. This is where Hera improves the
developer experience through its tailored classes and syntactic sugar to enable developers to easily orchestrate Python
functions. Check out [Hello World](hello-world.md) to get started!

If you want an introductory tour of Hera, check out the [Hera Tour](hera-tour.md)!
Loading

0 comments on commit 61bf73c

Please sign in to comment.