Merge branch 'master' into fix-gmolto

.flake8

@@ -0,0 +1,2 @@
+[flake8]
+exclude = examples

.github/ISSUE_TEMPLATE/bug_report.md

@@ -23,4 +23,5 @@ about: Create a report to help us improve
 
 ### Possible Solution
 
-Any thoughts as to potential solutions or ideas to go about finding one. Please include links to any research.
+Any thoughts as to potential solutions or ideas to go about finding one.
+Please include links to any research.

.github/workflows/sqaaas.yaml

@@ -0,0 +1,25 @@
+name: SQAaaS OSCAR
+
+on:
+  push:
+    branches: ["sqa"]
+    tags:
+      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10
+jobs:
+
+  sqaaas_job:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Add tox unit test step definition for a SQAaaS assessment
+        uses: eosc-synergy/sqaaas-step-action@v1
+        id: go_unit_test
+        with:
+          name: go_unit_test
+          container: "golang:1.21.4-alpine3.18"
+          tool: commands
+          commands: "go test ./... -v"
+
+      - name: SQAaaS assessment step
+        uses: eosc-synergy/sqaaas-assessment-action@v2
+        with:
+          qc_uni_steps: go_unit_test

.github/workflows/tests.yaml

@@ -8,16 +8,21 @@ jobs:
     runs-on: ubuntu-20.04
     steps:
     - name: Checkout
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: Setup Go
-      uses: actions/setup-go@v3
+      uses: actions/setup-go@v5
       with:
         go-version: '1.21'
 
     - name: Run tests
       run: go test ./pkg/... -cover -coverprofile=profile.cov
 
+    - name: Run Gosec Security Scanner
+      uses: securego/gosec@master
+      with:
+        args: ./...
+
     - name: Report coverage
       uses: codacy/codacy-coverage-reporter-action@v1
       with:

CODE_OF_CONDUCT.md

@@ -14,19 +14,22 @@ orientation.
 Examples of behavior that contributes to creating a positive environment
 include:
 
--   Using welcoming and inclusive language
--   Being respectful of differing viewpoints and experiences
--   Gracefully accepting constructive criticism
--   Focusing on what is best for the community
--   Showing empathy towards other community members
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
--   The use of sexualized language or imagery and unwelcome sexual attention or advances
--   Trolling, insulting/derogatory comments, and personal or political attacks
--   Public or private harassment
--   Publishing others' private information, such as a physical or electronic address, without explicit permission
--   Other conduct which could reasonably be considered inappropriate in a professional setting
+- The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
 
 ## Our Responsibilities
 
@@ -52,7 +55,7 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at products@grycap.upv.es. All
+reported by contacting the project team at <products@grycap.upv.es>. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.

README.md

@@ -1,7 +1,7 @@
 # OSCAR - Open Source Serverless Computing for Data-Processing Applications
 
-[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar)](https://goreportcard.com/report/github.com/grycap/oscar)
-[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://www.codacy.com/gh/grycap/oscar/dashboard?utm_source=github.com&utm_medium=referral&utm_content=grycap/oscar&utm_campaign=Badge_Coverage)
+[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar/v3)](https://goreportcard.com/report/github.com/grycap/oscar/v3)
+[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://app.codacy.com/gh/grycap/oscar/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 [![tests](https://github.com/grycap/oscar/actions/workflows/tests.yaml/badge.svg?branch=master)](https://github.com/grycap/oscar/actions/workflows/tests.yaml)
 [![build](https://github.com/grycap/oscar/workflows/build/badge.svg)](https://github.com/grycap/oscar/actions?query=workflow%3Abuild)
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/grycap/oscar)](https://github.com/grycap/oscar/pkgs/container/oscar)
@@ -107,7 +107,12 @@ license text.
 
 This development is partially funded by the [EGI Strategic and Innovation Fund](https://www.egi.eu/about/egi-council/egi-strategic-and-innovation-fund/).
 
-Partially funded by the project [AI-SPRINT](https://ai-sprint-project.eu) "AI in Secure Privacy-Preserving Computing Continuum" that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 101016577.
+Partially funded by the projects:
+
+- [AI-SPRINT](https://ai-sprint-project.eu) "AI in Secure Privacy-Preserving Computing Continuum" that has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant 101016577.
+- [interTwin](https://intertwin.eu) "An interdisciplinary Digital Twin Engine for science" that has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.
+- [AI4EOSC](https://ai4eosc.eu) "Artificial Intelligence for the European Open Science Cloud" that has received funding from the European Union’s Horizon Europe Research and Innovation Programme under Grant 101058593."
+- [iMagine](http://imagine-ai.eu) "AI-based image data analysis tools for aquatic research" that has received funding from the European Union’s Horizon Europe Research and Innovation Programme under Grant 101058625.
 
 Also, Grant PDC2021-120844-I00 funded by Ministerio de Ciencia e Innovación/Agencia Estatal de Investigación/ 10.13039/501100011033 and by “European Union NextGenerationEU/PRTR” and Grant PID2020-113126RB-I00 funded by Ministerio de Ciencia e Innovación/Agencia Estatal de Investigación/ 10.13039/501100011033.
 

deploy/ansible/README.md

@@ -1,3 +1,3 @@
 # Ansible playbook to deploy K3s and the OSCAR platform
 
-Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.
+Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.

docs/api.md

@@ -4,4 +4,8 @@ OSCAR exposes a secure REST API available at the Kubernetes master's node IP
 through an Ingress Controller. This API has been described following the
 [OpenAPI Specification](https://www.openapis.org/) and it is available below.
 
+> ℹ️
+>
+> The bearer token used to run a service can be either the OSCAR [service access token](invoking-sync.md#service-access-tokens) or the [user's Access Token](integration-egi.md#obtaining-an-access-token) if the OSCAR cluster is integrated with EGI Check-in.
+
 !!swagger api.yaml!!

docs/api.yaml

@@ -33,6 +33,7 @@ paths:
       description: List all created services
       security:
         - basicAuth: []
+        - token: []
       tags:
         - services
     post:
@@ -50,6 +51,7 @@ paths:
       description: Create a service
       security:
         - basicAuth: []
+        - token: []
       requestBody:
         content:
           application/json:
@@ -74,6 +76,7 @@ paths:
       description: Update a service
       security:
         - basicAuth: []
+        - token: []
       requestBody:
         content:
           application/json:
@@ -108,6 +111,7 @@ paths:
       operationId: ReadService
       security:
         - basicAuth: []
+        - token: []
       description: Read a service
     delete:
       summary: Delete service
@@ -124,6 +128,7 @@ paths:
       description: Delete a service
       security:
         - basicAuth: []
+        - token: []
       tags:
         - services
   '/system/logs/{serviceName}':
@@ -157,6 +162,7 @@ paths:
       operationId: ListJobs
       security:
         - basicAuth: []
+        - token: []
       description: List all jobs with their status
     delete:
       summary: Delete jobs
@@ -173,6 +179,7 @@ paths:
       description: Delete all jobs from a service.
       security:
         - basicAuth: []
+        - token: []
       parameters:
         - schema:
             type: boolean
@@ -214,6 +221,7 @@ paths:
       description: Get the logs from a job
       security:
         - basicAuth: []
+        - token: []
       parameters:
         - schema:
             type: boolean
@@ -234,6 +242,7 @@ paths:
       description: Delete a job
       security:
         - basicAuth: []
+        - token: []
       tags:
         - logs
   /system/info:
@@ -256,6 +265,7 @@ paths:
       description: Get system info
       security:
         - basicAuth: []
+        - token: []
   /health:
     get:
       summary: Health
@@ -316,6 +326,7 @@ paths:
       description: Get system configuration
       security:
         - basicAuth: []
+        - token: []
   '/run/{serviceName}':
     parameters:
       - schema:
@@ -607,4 +618,6 @@ servers:
   - url: 'https://localhost'
     description: 'Local testing'
   - url: 'https://inference.cloud.ai4eosc.eu'
-    description: 'AI4EOSC OSCAR cluster'
+    description: 'AI4EOSC OSCAR cluster'
+  - url: 'https://inference-walton.cloud.imagine-ai.eu'
+    description: 'iMagine OSCAR cluster'

docs/integration-egi.md

@@ -67,7 +67,7 @@ grant access for all users from that VO.
 
 The static web interface of OSCAR has been integrated with EGI Check-in and
 published in [ui.oscar.grycap.net](https://ui.oscar.grycap.net) to facilitate
-the authorization of users. To login through EGI Checkín using OIDC tokens,
+the authorization of users. To login through EGI Check-In using OIDC tokens,
 users only have to put the endpoint of its OSCAR cluster and click on the
 "EGI CHECK-IN" button.
 
@@ -87,3 +87,18 @@ create a new account configuration for the
 After that, clusters can be
 added with the command [`oscar-cli cluster add`](oscar-cli.md#add) specifying
 the oidc-agent account name with the `--oidc-account-name` flag.
+
+### Obtaining an Access Token
+
+Once logged in via EGI Check-In you can obtain an Access Token with one of this approaches:
+
+* From the command-line, using `oidc-agent` with the following command:
+
+    ```sh
+    oidc-token <account-short-name>
+    ```
+    where `account-short-name` is the name of your account configuration.
+
+* From the EGI Check-In Token Portal: [https://aai.egi.eu/token](https://aai.egi.eu/token)
+
+![egi-checkin-token-portal.png](images/oidc/egi-checkin-token-portal.png)

docs/invoking-async.md

@@ -2,11 +2,13 @@
 
 For event-driven file processing, OSCAR automatically manages the creation
 and [notification system](https://docs.min.io/minio/baremetal/monitoring/bucket-notifications/bucket-notifications.html#minio-bucket-notifications)
-of MinIO buckets in order to allow the event-driven invocation of services
-using asynchronous requests, generating a Kubernetes job for every file to be
-processed.
-
+of MinIO buckets. This allow the event-driven invocation of services
+using asynchronous requests for every file uploaded to the bucket, which generates a Kubernetes job for every file to be processed.
 
 ![oscar-async.png](images/oscar-async.png)
 
+These jobs will be queued up in the Kubernetes scheduler and will be processed whenever there are resources available. If OSCAR cluster has been deployed as an elastic Kubernetes cluster (see [Deployment with IM](https://docs.oscar.grycap.net/deploy-im-dashboard/)), then new Virtual Machines will be provisioned (up to the maximum number of nodes defined) in the underlying Cloud platform and seamlessly integrated in the Kubernetes clusters to proceed with the execution of jobs. These nodes will be terminated as the worload is reduced. Notice that the output files can be stores in MinIO or in any other storage back-end supported by the [FaaS supervisor](oscar-service.md#faas-supervisor). 
+
+ Note that if your OSCAR service runs an AI model for inference, each job will load the AI model weights before performing the inference. You can mitigate this penalty by adjusting the inference code to process a compressed file with several images.
 
+If you want to process a large number of data files, consider using [OSCAR Batch](https://github.com/grycap/oscar-batch), a tool designed to perform batch-based processing in OSCAR clusters. It includes a coordinator tool where the user provides a MinIO bucket containing files for processing. This service calculates the optimal number of parallel service invocations that can be accommodated within the cluster, according to its current status, and distributes the image processing workload accordingly among the service invocations. This is mainly intended to process large amounts of files, for example, historical data.

docs/invoking-sync.md

@@ -83,8 +83,8 @@ base64 input.png | curl -X POST -H "Authorization: Bearer <TOKEN>" \
 
 ## Service access tokens
 
-As detailed in the [API specification](api.md), invocation paths require the
-service access token in the request header for authentication. Service access
+As detailed in the [API specification](api.md), invocation paths require either the
+service access token or the Access Token of the user when the cluster is integrated with EGI Check-in, in the request header for authentication (any of them is valid). Service access
 tokens are auto-generated in service creation and update, and MinIO eventing
 system is automatically configured to use them for event-driven file
 processing. Tokens can be obtained through the API, using the

docs/invoking.md

@@ -2,7 +2,18 @@
 
 OSCAR services can be executed:
 
-  - [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained. Useful for short-lived service invocations.
+  - [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained. 
   - [Asynchronously](invoking-async.md), typically in response to a file upload to MinIO or via the OSCAR API.
-  - As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. a Jupyter Notebook)
+  - As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. Jupyter Notebook)
+
+
+After reading the different service execution types, take into account the following considerations to better decide the most appropriate execution type for your use case:
+
+* **Scalability**: Asynchronous invocations provide the best throughput when dealing with multiple concurrent data processing requests, since these are processed by independent jobs which are managed by the Kubernetes scheduler. A two-level elasticity approach is used (increase in the number of pods and increase in the number of Virtual Machines, if the OSCAR cluster was configured to be elastic). This is the recommended approach when each processing request exceeds the order of tens of seconds.
+
+* **Reduced Latency** Synchronous invocations are oriented for short-lived (< tens of seconds) bursty requests. A certain number of containers can be configured to be kept alive to avoid the performance penalty of spawning new ones while providing an upper bound limit (see [`min_scale` and `max_scale` in the FDL](fdl.md#synchronoussettings), at the expense of always consuming resources in the OSCAR cluster. If the processing file is in the order of several MBytes it may not fit in the payload of the HTTP request.
+
+* **Easy Access** For services that provide their own user interface or their own API, exposed services provide the ability to execute them in OSCAR and benefit for an auto-scaled configuration in case they are [stateless](https://en.wikipedia.org/wiki/Service_statelessness_principle). This way, users can directly access the service using its well-known interfaces by the users. 
+
+If you want to process a large number of files, please consider using [OSCAR-batch](https://github.com/grycap/oscar-batch).
 

docs/minio-usage.md

@@ -0,0 +1,50 @@
+# Using the MinIO Storage Provider
+
+Each OSCAR cluster includes a deployed MinIO instance, which is used to trigger service executions. When a service is configured to use MinIO as its storage provider, it monitors a specified input folder for new data. Whenever new data is added to this folder, it triggers the associated service to execute. 
+
+## Using graphical interfaces
+
+These folders can be accessed via both the OSCAR UI and the MinIO console UI.
+
+- **Using OSCAR-UI**: The following image highlights the section where MinIO buckets are accessible. Users can view a list of available buckets and perform operations such as uploading and deleting files.
+
+![minio-buckets](images/bucket-list.png)
+
+- **Using the MinIO Console UI**: Access details for this interface are available in the "Info" tab within the OSCAR UI. This tab provides the MinIO console endpoint and the necessary credentials to log in, where the *Access Key* serves as the username, and the *Secret Key* functions as the password.
+
+![oscar-info](images/oscar-info.png)
+
+Finally, the following image provides an overview of the MinIO login panel and the "Object Browser" tab. Once logged in, the "Object Browser" tab allows users to navigate their available buckets, view stored objects, and perform various operations such as uploading, downloading, or deleting files.
+
+![oscar-info](images/minio-ui.png)
+
+Further information about the MinIO Console avaliable on [MinIO Console documentation](https://min.io/docs/minio/linux/administration/minio-console.html).
+
+## Using command-line interfaces
+
+MinIO buckets can also be managed through [oscar-cli command-line](https://github.com/grycap/oscar-cli) or the official [MinIO client](https://min.io/docs/minio/linux/reference/minio-mc.html). 
+
+- **oscar-cli**: The OSCAR client provides a dedicated set of commands for accessing files within buckets. It is important to note that this interface does not support DELETE or UPDATE operations. Below is a brief overview of the available commands and their functionalities.
+  - [get-file](https://docs.oscar.grycap.net/oscar-cli/#get-file): Get file from a service's storage provider.
+  - [list-files](https://docs.oscar.grycap.net/oscar-cli/#list-files): List files from a service's storage provider path.
+  - [put-file](https://docs.oscar.grycap.net/oscar-cli/#put-file): Upload a file on a service storage provider.
+
+  An example of a put-file operation:
+
+  ``` bash
+  oscar-cli service put-file fish-detector.yaml minio .path/to/your/images ./fish-detector/input/
+  ```
+
+- **mc**: If a user wants to use the MinIO client it needs to follow some previous steps.
+  - *Install the client*: Detailed instructions for installing the MinIO client (mc) are available in [the official documentation](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc).
+  - *Configure the MinIO instance*: The client requires credentials to connect and interact with the MinIO instance. This configuration can be set with the following command:
+
+    ``` bash
+    mc alias set myminio https://minio.gracious-varahamihira6.im.grycap.net YOUR-ACCESS-KEY YOUR-SECRET-KEY
+    ```
+
+  Once the client is configured, users can perform various operations supported by the MinIO client. For a complete list of available commands and their usage, refer to the [MinIO client reference](https://min.io/docs/minio/linux/reference/minio-mc.html#command-quick-reference). The following example demonstrates a PUT operation, where a file is uploaded to a specific folder within a bucket.
+
+  ```bash
+  mc cp /path/to/your/images/*.jpg myminio/fish-detector/input/
+  ```