diff --git a/ogc-api-processes-with-zoo/README.md b/ogc-api-processes-with-zoo/README.md index 37fa371..a450eb2 100644 --- a/ogc-api-processes-with-zoo/README.md +++ b/ogc-api-processes-with-zoo/README.md @@ -1,6 +1,156 @@ +# ZOO-Project OGC API Processes Deployment +This directory contains the deployment configuration for ZOO-Project with OGC API Processes using Skaffold and Helm. +## Prerequisites + +Before deploying, ensure you have the following tools installed: +- **kubectl** - Kubernetes command-line tool +- **helm** (v3+) - Kubernetes package manager +- **skaffold** - Kubernetes development tool + +Add the required Helm repositories: + +```bash +helm repo add zoo-project https://zoo-project.github.io/charts/ +helm repo add localstack https://helm.localstack.cloud +helm repo update +``` + +## Deployment Profiles + +### Standard Installation + +Deploy ZOO-Project with Calrissian workflow engine: + +```bash +skaffold dev +``` + +This profile includes: +- ZOO-Project DRU (v0.8.2) +- Calrissian CWL runner +- LocalStack S3 for storage +- Code-server development environment +- RabbitMQ message queue +- PostgreSQL database +- Redis cache + +**Access points:** +- Code-server: http://localhost:8000 +- ZOO-Project API: http://localhost:8080 +- WebSocket: http://localhost:8888 + +### KEDA Autoscaling Profile + +Deploy with Kubernetes Event-Driven Autoscaling (KEDA) and Kyverno policy enforcement: + +```bash +skaffold dev -p keda +``` + +Additional features: +- **KEDA autoscaling** based on PostgreSQL and RabbitMQ metrics +- **Kyverno** policy engine for pod protection +- **Eviction controller** to protect active workers from termination +- Automatic scaling of ZOO-FPM workers based on queue depth + +This profile is ideal for production environments requiring dynamic scaling. + +### Argo Workflows Profile + +Deploy with Argo Workflows for advanced workflow orchestration: + +```bash +# Create S3 credentials secret first +kubectl create secret generic s3-service -n eoap-zoo-project \ + --from-literal=rootUser=test \ + --from-literal=rootPassword=test \ + --dry-run=client -o yaml | kubectl apply -f - + +# Deploy with Argo profile +skaffold dev -p argo +``` + +Additional features: +- **Argo Workflows** (v3.7.1) for workflow orchestration +- Workflow artifact storage in LocalStack S3 +- Namespaced deployment with instance isolation +- Workflow TTL and pod garbage collection +- Argo Workflows UI for workflow visualization + +**Additional access points:** +- Argo Workflows UI: http://localhost:2746 +- LocalStack S3: http://localhost:9000 + +### macOS / ARM Processor Support + +For Apple Silicon (M1/M2) or other ARM-based systems: + +```bash +skaffold dev -p macos +``` + +This profile configures `hostpath` storage class compatible with Docker Desktop on macOS. + +## Cleanup + +When switching between profiles or redeploying, use the cleanup script to ensure all resources are properly removed: + +```bash +./cleanup.sh ``` -helm repo add zoo-project https://zoo-project.github.io/charts/ -helm repo add localstack https://helm.localstack.cloud -``` \ No newline at end of file + +The cleanup script will: +- Stop running Skaffold processes +- Remove Helm releases (ZOO-Project, Kyverno, LocalStack) +- Clean up KEDA and Argo Workflows resources +- Remove Custom Resource Definitions (CRDs) +- Force removal of stuck namespaces and persistent volumes +- Validate complete cleanup + +**Note:** This script is particularly important when switching between KEDA and Argo profiles to avoid resource conflicts. + +## Combining Profiles + +Profiles can be combined for specific deployment scenarios: + +```bash +# KEDA + macOS +skaffold dev -p keda,macos + +# Argo + macOS +skaffold dev -p argo,macos +``` + +## Troubleshooting + +### Namespace stuck in Terminating state +Run the cleanup script which handles finalizer removal: +```bash +./cleanup.sh +``` + +### Port conflicts +Ensure no other services are using the default ports (8000, 8080, 8888, 2746, 9000). + +### Persistent Volume issues +The cleanup script removes all PVs. If issues persist, manually check: +```bash +kubectl get pv +kubectl delete pv --grace-period=0 --force +``` + +## Configuration Files + +- **skaffold.yaml** - Main deployment configuration with all profiles +- **values.yaml** - Default Helm values for standard/KEDA deployments +- **values_argo.yaml** - Helm values for Argo Workflows deployment +- **cleanup.sh** - Resource cleanup script + +## More Information + +For detailed information about ZOO-Project, visit: +- [ZOO-Project Documentation](https://zoo-project.github.io/docs/) +- [ZOO-Project Helm Charts](https://github.com/ZOO-Project/charts) + diff --git a/ogc-api-processes-with-zoo/cleanup.sh b/ogc-api-processes-with-zoo/cleanup.sh new file mode 100755 index 0000000..d43fcb9 --- /dev/null +++ b/ogc-api-processes-with-zoo/cleanup.sh @@ -0,0 +1,124 @@ +#!/bin/bash +set -e + +echo "๐Ÿงน Complete cluster cleanup..." + +# 1. Stop skaffold if running +echo "โน๏ธ Stopping skaffold..." +pkill -f "skaffold dev" || true +sleep 2 + +# 2. Remove Helm releases (without waiting) +echo "๐Ÿ—‘๏ธ Removing Helm releases..." +helm uninstall zoo-project-dru -n eoap-zoo-project --no-hooks --timeout 10s 2>/dev/null || true +helm uninstall eoap-zoo-project-coder -n eoap-zoo-project --no-hooks --timeout 10s 2>/dev/null || true +helm uninstall eoap-zoo-project-localstack -n eoap-zoo-project --no-hooks --timeout 10s 2>/dev/null || true +helm uninstall kyverno -n kyverno-system --no-hooks --timeout 10s 2>/dev/null || true + +# 3. Remove Kyverno webhooks +echo "๐Ÿ”Œ Remove Kyverno webhooks..." +kubectl delete validatingwebhookconfigurations -l app.kubernetes.io/part-of=kyverno --ignore-not-found --wait=false || true +kubectl delete mutatingwebhookconfigurations -l app.kubernetes.io/part-of=kyverno --ignore-not-found --wait=false || true + +# 4. Remove residual KEDA resources +echo "๐Ÿงฝ Removing residual KEDA resources..." +for r in $(kubectl get scaledobjects.keda.sh -A -o name 2>/dev/null); do + kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true + kubectl delete "$r" --wait=false 2>/dev/null || true +done + +for r in $(kubectl get triggerauthentications.keda.sh -A -o name 2>/dev/null); do + kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true + kubectl delete "$r" --wait=false 2>/dev/null || true +done + +# 4b. Remove residual Argo Workflows resources +echo "๐Ÿงฝ Removing residual Argo Workflows resources..." +for r in $(kubectl get workflows.argoproj.io -A -o name 2>/dev/null); do + kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true + kubectl delete "$r" --wait=false 2>/dev/null || true +done + +for r in $(kubectl get workflowtemplates.argoproj.io -A -o name 2>/dev/null); do + kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true + kubectl delete "$r" --wait=false 2>/dev/null || true +done + +for r in $(kubectl get cronworkflows.argoproj.io -A -o name 2>/dev/null); do + kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true + kubectl delete "$r" --wait=false 2>/dev/null || true +done + +# 5. Remove CRDs with finalizers +echo "๐Ÿ—‚๏ธ Removing KEDA, Kyverno, and Argo CRDs..." + +# First remove the resource-policy annotation that prevents deletion +for crd in workflows.argoproj.io workflowtemplates.argoproj.io cronworkflows.argoproj.io clusterworkflowtemplates.argoproj.io workfloweventbindings.argoproj.io workflowartifactgctasks.argoproj.io workflowtasksets.argoproj.io workflowtaskresults.argoproj.io; do + if kubectl get crd "$crd" >/dev/null 2>&1; then + kubectl annotate crd "$crd" helm.sh/resource-policy- 2>/dev/null || true + kubectl patch crd "$crd" --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true + kubectl delete crd "$crd" --ignore-not-found 2>/dev/null || true + fi +done + +for c in $(kubectl get crd -o name 2>/dev/null | grep -E 'keda.sh|kyverno.io'); do + kubectl patch "$c" --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true + kubectl delete "$c" --wait=false 2>/dev/null || true +done + +sleep 3 + +# 6. Remove pods and PVCs +echo "๐Ÿ’พ Removing pods and PVCs..." +kubectl delete pods -n eoap-zoo-project --all --force --grace-period=0 2>/dev/null || true +sleep 2 + +# Remove PVCs with finalizers +for pvc in $(kubectl get pvc -n eoap-zoo-project -o name 2>/dev/null); do + kubectl patch -n eoap-zoo-project "$pvc" --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true + kubectl delete -n eoap-zoo-project "$pvc" --wait=false 2>/dev/null || true +done + +sleep 2 + +# Remove ALL PVs (not just those with eoap-zoo-project in the name) +for pv in $(kubectl get pv -o name 2>/dev/null); do + kubectl patch "$pv" --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true + kubectl delete "$pv" --wait=false 2>/dev/null || true +done + +# 7. Remove namespaces +echo "๐Ÿ—‘๏ธ Removing namespaces..." +kubectl delete ns eoap-zoo-project --wait=false 2>/dev/null || true +kubectl delete ns kyverno-system --wait=false 2>/dev/null || true + +# 8. Force finalization of stuck namespaces +echo "โšก Forcing finalization of namespaces..." +for ns in eoap-zoo-project kyverno-system; do + if kubectl get ns "$ns" >/dev/null 2>&1; then + kubectl get ns "$ns" -o json | jq '.spec.finalizers=[]' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f - 2>/dev/null || true + fi +done + +# 9. Wait for everything to be cleaned up +echo "โณ Waiting for complete cleanup..." +sleep 10 + +# 10. Check final status +echo "โœ… Checking cluster status..." +echo "" +echo "Remaining namespaces:" +kubectl get ns | grep -E 'kyverno-system|eoap-zoo-project' || echo " โœ“ Namespaces cleaned up" + +echo "" +echo "Remaining CRDs:" +kubectl get crd 2>/dev/null | grep -E 'keda.sh|kyverno.io|argoproj.io' || echo " โœ“ CRDs cleaned up" +echo "" +echo "Remaining PVs:" +kubectl get pv 2>/dev/null || echo " โœ“ No PVs" + +echo "" +echo "โœจ Cleanup complete!" +echo "" +echo "To deploy, run:" +echo " skaffold dev -p keda" diff --git a/ogc-api-processes-with-zoo/skaffold.yaml b/ogc-api-processes-with-zoo/skaffold.yaml index 10cbf1b..9be4198 100644 --- a/ogc-api-processes-with-zoo/skaffold.yaml +++ b/ogc-api-processes-with-zoo/skaffold.yaml @@ -2,7 +2,10 @@ apiVersion: skaffold/v4beta9 kind: Config metadata: name: eoap-zoo-project - + +build: + platforms: ["linux/amd64"] + deploy: helm: releases: @@ -10,26 +13,30 @@ deploy: remoteChart: zoo-project/zoo-project-dru namespace: eoap-zoo-project createNamespace: true - version: 0.3.23 + version: 0.8.2 valuesFiles: - values.yaml setValues: iam.enabled: false - cookiecutter.templateUrl: https://github.com/eoap/zoo-service-template.git - cookiecutter.templateBranch: feature-collection + cookiecutter.templateUrl: https://github.com/GeoLabs/zoo-service-template.git + cookiecutter.templateBranch: feature/eoap-tooling filter_in.enabled: true filter_out.enabled: true persistence.procServicesAccessMode: ReadWriteMany - customConfig.main.additional_parameters: |- - s3_bucket=results - region_name=us-east-1 - aws_secret_access_key=test - aws_access_key_id=test - endpoint_url=http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 - #customConfig.main.pod_env_vars: |- - # C="1" - #customConfig.main.pod_node_selector: |- - # "kubernetes.io/hostname"=minikube + websocketd.enabled: true + websocketd.port: 8888 + websocketd.image.repository: zooproject/websocketd + websocketd.image.tag: 67449315857b54bbc970f02c7aa4fd10a94721f0 + websocketd.image.pullPolicy: IfNotPresent + rabbitmq.autoSetup.enabled: true + redis.enabled: true + workflow.additionalInputs: + s3_bucket: results + region_name: us-east-1 + aws_secret_access_key: test + aws_access_key_id: test + endpoint_url: http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 + - name: eoap-zoo-project-coder chartPath: ../charts/coder namespace: eoap-zoo-project @@ -37,14 +44,13 @@ deploy: setValues: coder.coderImage: eoepca/pde-code-server:1.0.0 coder.workspace: ogc-api-processes-with-zoo - coderstorageClassName: standard coder.workspaceStorage: 10Gi coderResources.limits.cpu: '2' coderResources.limits.memory: '6442450944' coderResources.requests.cpu: '1' coderResources.requests.memory: '4294967296' calrissian.enabled: true - + setFiles: { initScript: ./files/init.sh, bashrcScript: ./files/bash-rc, @@ -57,6 +63,140 @@ deploy: setValues: service.type: ClusterIP +profiles: + + - name: macos + + patches: + - op: add + path: /deploy/helm/releases/0/setValues/zoofpm.image.pullPolicy + value: Never + - op: add + path: /deploy/helm/releases/0/setValues/zookernel.image.pullPolicy + value: Never + - op: add + path: /deploy/helm/releases/0/setValues/workflow.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/workflow.argo.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.storageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.procServicesStorageClass + value: hostpath + - op: add + path: /deploy/helm/releases/0/setValues/persistence.tmpStorageClass + value: hostpath + - op: add + path: /deploy/helm/releases/1/setValues/coder.storageClassName + value: hostpath + - op: add + path: /deploy/helm/releases/1/setValues/calrissian.storageClassName + value: hostpath + + - name: keda + + patches: + # Install Kyverno FIRST so CRDs exist before applying policies + - op: add + path: /deploy/helm/releases/0 + value: + name: kyverno + remoteChart: kyverno/kyverno + version: "3.5.2" + namespace: kyverno-system + wait: true + createNamespace: true + setValues: + crds.install: true + + # Enable KEDA + protections on zoo-project-dru (now at index 1) + - op: add + path: /deploy/helm/releases/1/setValues/keda.enabled + value: true + - op: add + path: /deploy/helm/releases/1/setValues/keda.skipScaledObject + value: false + - op: add + path: /deploy/helm/releases/1/setValues/keda.triggers.postgresql.enabled + value: true + - op: add + path: /deploy/helm/releases/1/setValues/keda.triggers.postgresql.useConfigMap + value: false + - op: add + path: /deploy/helm/releases/1/setValues/keda.triggers.postgresql.query + value: "SELECT COUNT(*) FROM workers WHERE status = 1" + - op: add + path: /deploy/helm/releases/1/setValues/keda.triggers.rabbitmq.enabled + value: true + + # Kyverno protections (requires Kyverno CRDs installed above) + - op: add + path: /deploy/helm/releases/1/setValues/keda.kyverno.enabled + value: false + - op: add + path: /deploy/helm/releases/1/setValues/keda.kyverno.policies.zoofpmProtection.enabled + value: true + - op: add + path: /deploy/helm/releases/1/setValues/keda.kyverno.policies.zoofpmProtection.failurePolicy + value: Enforce + + # Eviction Controller to protect active workers + - op: add + path: /deploy/helm/releases/1/setValues/keda.evictionController.enabled + value: true + - op: add + path: /deploy/helm/releases/1/setValues/keda.evictionController.image.repository + value: ghcr.io/zoo-project/zoofpm-eviction-controller + - op: add + path: /deploy/helm/releases/1/setValues/keda.evictionController.image.tag + value: latest + + - name: argo + + patches: + - op: replace + path: /deploy/helm/releases/0/valuesFiles + value: ["values_argo.yaml"] + - op: replace + path: /deploy/helm/releases/0/setValues/cookiecutter.templateUrl + value: https://github.com/gfenoy/zoo-argo-wf-proc-service-template.git + - op: replace + path: /deploy/helm/releases/0/setValues/cookiecutter.templateBranch + value: feature/use-argo-wf-namespace + + portForward: + # Argo Workflows Server + - resourceType: service + resourceName: zoo-project-dru-argo-workflows-server + namespace: eoap-zoo-project + port: 2746 + localPort: 2746 + # ZOO-Project services + - resourceType: service + resourceName: zoo-project-dru-service + namespace: eoap-zoo-project + port: 80 + localPort: 8080 + - resourceType: service + resourceName: zoo-project-dru-websocketd + namespace: eoap-zoo-project + port: 8888 + localPort: 8888 + # LocalStack S3 + - resourceType: service + resourceName: eoap-zoo-project-localstack + namespace: eoap-zoo-project + port: 4566 + localPort: 9000 + # Code-server + - resourceType: service + resourceName: code-server-service + namespace: eoap-zoo-project + port: 8080 + localPort: 8000 portForward: - resourceType: service @@ -64,7 +204,12 @@ portForward: namespace: eoap-zoo-project address: localhost port: 8080 - localPort: 8000 + localPort: 8000 + - resourceType: service + resourceName: zoo-project-dru-websocketd + namespace: eoap-zoo-project + port: 8888 + localPort: 8888 - resourceType: service resourceName: zoo-project-dru-service namespace: eoap-zoo-project diff --git a/ogc-api-processes-with-zoo/values.yaml b/ogc-api-processes-with-zoo/values.yaml index 054642a..a67358b 100644 --- a/ogc-api-processes-with-zoo/values.yaml +++ b/ogc-api-processes-with-zoo/values.yaml @@ -106,16 +106,30 @@ files: cwlVersion: v1.0 class: CommandLineTool - id: main - inputs: {} - outputs: {} + id: stage-in-directory + inputs: + reference: + type: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI + doc: "A STAC Item to stage" + label: "STAC Item URL" + outputs: + staged: + type: Directory + outputBinding: + glob: . baseCommand: - python - stage.py arguments: - - $( inputs.input ) + - $( inputs.reference.value ) + # - $( inputs.another_input ) # This is an additional input to demonstrate the use of multiple inputs requirements: + NetworkAccess: + networkAccess: true + SchemaDefRequirement: + types: + - $import: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml DockerRequirement: dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0 InlineJavascriptRequirement: {} @@ -132,23 +146,23 @@ files: config = stac_asset.Config(warn=True) async def main(href: str): - + item = pystac.read_file(href) - + os.makedirs(item.id, exist_ok=True) cwd = os.getcwd() - + os.chdir(item.id) item = await stac_asset.download_item(item=item, directory=".", config=config) os.chdir(cwd) - + cat = pystac.Catalog( id="catalog", description=f"catalog with staged {item.id}", title=f"catalog with staged {item.id}", ) cat.add_item(item) - + cat.normalize_hrefs("./") cat.save(catalog_type=pystac.CatalogType.SELF_CONTAINED) @@ -178,28 +192,50 @@ files: type: string endpoint_url: type: string + stac_catalog: + doc: "The folder containing the STAC catalog to stage out" + label: "STAC Catalog folder" + type: Directory outputs: s3_catalog_output: + type: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI outputBinding: - outputEval: ${ return "s3://" + inputs.s3_bucket + "/" + inputs.sub_path + "/catalog.json"; } - type: string + glob: catalog-uri.txt + loadContents: true + outputEval: | + ${ + return { "value": self[0].contents, "type": "https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI" }; + } + stdout: catalog-uri.txt baseCommand: - python - stage.py arguments: - - $( inputs.wf_outputs.path ) + - $( inputs.stac_catalog.path ) - $( inputs.s3_bucket ) - - $( inputs.sub_path ) + - ${ + var firstPart = (Math.random() * 46656) | 0; + var secondPart = (Math.random() * 46656) | 0; + firstPart = ("000" + firstPart.toString(36)).slice(-3); + secondPart = ("000" + secondPart.toString(36)).slice(-3); + return inputs.sub_path + "-" + firstPart + secondPart; + } + requirements: + NetworkAccess: + networkAccess: true + SchemaDefRequirement: + types: + - $import: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml DockerRequirement: dockerPull: ghcr.io/eoap/mastering-app-package/stage:1.0.0 InlineJavascriptRequirement: {} EnvVarRequirement: envDef: - AWS_ACCESS_KEY_ID: $( inputs.aws_access_key_id ) - AWS_SECRET_ACCESS_KEY: $( inputs.aws_secret_access_key ) - AWS_REGION: $( inputs.region_name ) - AWS_S3_ENDPOINT: $( inputs.endpoint_url ) + aws_access_key_id: $( inputs.aws_access_key_id ) + aws_secret_access_key: $( inputs.aws_secret_access_key ) + aws_region_name: $( inputs.region_name ) + aws_endpoint_url: $( inputs.endpoint_url ) ResourceRequirement: {} InitialWorkDirRequirement: listing: @@ -224,10 +260,10 @@ files: print(f"bucket: {bucket}", file=sys.stderr) print(f"subfolder: {subfolder}", file=sys.stderr) - aws_access_key_id = os.environ["AWS_ACCESS_KEY_ID"] - aws_secret_access_key = os.environ["AWS_SECRET_ACCESS_KEY"] - region_name = os.environ["AWS_REGION"] - endpoint_url = os.environ["AWS_S3_ENDPOINT"] + aws_access_key_id = os.environ["aws_access_key_id"] + aws_secret_access_key = os.environ["aws_secret_access_key"] + region_name = os.environ["aws_region_name"] + endpoint_url = os.environ["aws_endpoint_url"] shutil.copytree(cat_url, "/tmp/catalog") cat = pystac.read_file(os.path.join("/tmp/catalog", "catalog.json")) @@ -334,4 +370,56 @@ files: print(f"upload catalog.json to s3://{bucket}/{subfolder}", file=sys.stderr) pystac.write_file(cat, cat.get_self_href()) - print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) \ No newline at end of file + print(f"s3://{bucket}/{subfolder}/catalog.json", file=sys.stdout) + + stagein-file.yaml: | + cwlVersion: v1.0 + + class: CommandLineTool + id: stage-in-file + + inputs: + reference: + type: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml#URI + doc: "An URL to stage" + label: "Reference URL" + outputs: + staged: + type: File + outputBinding: + glob: staged + baseCommand: + - python + - stage.py + arguments: + - $( inputs.reference.value ) + requirements: + NetworkAccess: + networkAccess: true + SchemaDefRequirement: + types: + - $import: https://raw.githubusercontent.com/eoap/schemas/main/string_format.yaml + DockerRequirement: + dockerPull: ghcr.io/eoap/application-package-patterns/vegetation-indexes:0.1.1 + InlineJavascriptRequirement: {} + InitialWorkDirRequirement: + listing: + - entryname: stage.py + entry: |- + import sys + import requests + import planetary_computer + + href = sys.argv[1] + + signed_url = planetary_computer.sign(href) + output_path = "staged" + + response = requests.get(signed_url, stream=True) + response.raise_for_status() # Raise an error for bad status codes + + with open(output_path, "wb") as f: + for chunk in response.iter_content(chunk_size=8192): + f.write(chunk) + + print(f"Downloaded to {output_path}") diff --git a/ogc-api-processes-with-zoo/values_argo.yaml b/ogc-api-processes-with-zoo/values_argo.yaml new file mode 100644 index 0000000..9ee4c9d --- /dev/null +++ b/ogc-api-processes-with-zoo/values_argo.yaml @@ -0,0 +1,232 @@ +# ZOO-Project-DRU Configuration with Argo Workflows for eoap-zoo-project + +argo: + enabled: true + installCRDs: true # Install CRDs automatically + instanceID: "eoap-zoo-project" + + # Images configuration + cwlwrapperImage: "eoepca/cwl-wrapper:0.12.1" + stageOutImage: "ghcr.io/eoap/mastering-app-package/stage:1.1.1" + + # Service Account + serviceAccount: + name: "argo-workflow" + + # Feature collection script + featureCollectionScript: "" + + # Argo Events configuration + events: + enabled: false + + autoTokenManagement: true + restartOnTokenUpdate: false + + # S3 configuration for artifact repository + s3: + bucket: "results" + endpoint: "eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566" + insecure: true + # LocalStack secret configuration + secretName: "s3-service" + accessKeySecretKey: "rootUser" + secretKeySecretKey: "rootPassword" + + # Ingress configuration for Argo Workflows UI + ingress: + enabled: false + +workflow: + argo: + # Deploy Argo workflows using the Helm chart + internal: true + enabled: true + instanceID: "eoap-zoo-project" + + # Common configuration + defaultVolumeSize: "12Gi" + defaultMaxRam: "2Gi" + wfServer: "http://zoo-project-dru-argo-workflows-server.eoap-zoo-project.svc.cluster.local:2746" + wfToken: "" + wfNamespace: "eoap-zoo-project" + wfSynchronizationCm: "semaphore-argo-cwl-runner-stage-in-out" + CwlRunnerTemplare: "argo-cwl-runner-stage-in-out" + CwlRunnerEndpoint: "calrissian-runner" + + additionalInputs: + s3_bucket: results + region_name: us-east-1 + aws_secret_access_key: test + aws_access_key_id: test + endpoint_url: http://eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 + +minio: + enabled: false # Using external localstack instead + +cookiecutter: + templateUrl: https://github.com/gfenoy/zoo-argo-wf-proc-service-template.git + templateBranch: main + +# CWL Wrapper configuration for Argo Workflows +wrapper: + rules: "" + main: "" + stageIn: "" + stageOut: "" + +iam: + enabled: false + +webui: + enabled: false + +websocketd: + enabled: true + port: 8888 + image: + repository: zooproject/websocketd + tag: 67449315857b54bbc970f02c7aa4fd10a94721f0 + pullPolicy: IfNotPresent + +redis: + enabled: true + +postgresql: + enabled: true + auth: + database: zoo + username: zoo + password: zoo-password + +rabbitmq: + autoSetup: + enabled: true + +# Configuration for Argo Workflows chart +argo-workflows: + fullnameOverride: "zoo-project-dru-argo-workflows" + singleNamespace: true + + # Image versions + images: + tag: "v3.7.1" + pullPolicy: IfNotPresent + + # Global artifact repository configuration + artifactRepository: + archiveLogs: true + s3: + endpoint: eoap-zoo-project-localstack.eoap-zoo-project.svc.cluster.local:4566 + bucket: results + insecure: true + accessKeySecret: + name: s3-service + key: rootUser + secretKeySecret: + name: s3-service + key: rootPassword + + # Workflow Controller Configuration + controller: + enabled: true + instanceID: + enabled: true + useReleaseName: false + explicitID: "eoap-zoo-project" + + # Metrics configuration + metricsConfig: + enabled: true + path: /metrics + port: 9090 + + serviceMonitor: + enabled: false + + extraArgs: + - --managed-namespace=eoap-zoo-project + - --namespaced + + clusterWorkflowTemplates: + enabled: false + + workflowDefaults: + spec: + serviceAccountName: argo-workflow + ttlStrategy: + secondsAfterCompletion: 3600 + secondsAfterSuccess: 300 + secondsAfterFailure: 3600 + podGC: + strategy: OnPodCompletion + deleteDelayDuration: 60s + + workflowTTLWorkers: 4 + podCleanupWorkers: 4 + + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + + # Argo Server Configuration + server: + enabled: true + serviceType: ClusterIP + servicePort: 2746 + authModes: + - server + secure: false + namespaced: true + + clusterWorkflowTemplates: + enabled: false + + extraArgs: + - --namespaced + - --managed-namespace=eoap-zoo-project + + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + + serviceMonitor: + enabled: false + + # RBAC Configuration + rbac: + create: true + + # Service Account Configuration + serviceAccount: + create: true + + # Disable CRD installation + crds: + install: false + keep: true + +# Disable monitoring to keep configuration simple +monitoring: + enabled: false + +argo-events: + enabled: false + +# Additional base configuration +filter_in: + enabled: true + +filter_out: + enabled: true + +persistence: + procServicesAccessMode: ReadWriteMany