Merge branch 'master' into v5.5.0_test

galaxyproject · Jul 5, 2024 · a79797c · a79797c
2 parents 2783ed0 + b8c9649
commit a79797c
Show file tree

Hide file tree

Showing 43 changed files with 851 additions and 132 deletions.
diff --git a/.github/workflows/packaging.yml b/.github/workflows/packaging.yml
@@ -20,7 +20,7 @@ jobs:
     name: Package and push from PR
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
            persist-credentials: false
       - uses: cloudve/helm-ci@master
@@ -36,7 +36,7 @@ jobs:
     name: Package and push manual invocation
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
            persist-credentials: false
       - uses: cloudve/helm-ci@master
@@ -47,3 +47,25 @@ jobs:
           charts-token: ${{ secrets.CHARTS_TOKEN }}
           github-labels: ${{ github.event.inputs.version-bump }}
           git-branch: ${{ github.event.inputs.branch-name }}
+  tag-and-release:
+    needs: [ package-from-pr, package-from-manual ]
+    name: Create a tag and GitHub release for this version.
+    runs-on: ubuntu-latest
+    env:
+      GH_TOKEN: ${{ github.token }}
+    if: |
+      always() 
+      && contains(needs.*.result, 'success')
+      && !contains(needs.*.result, 'failure')
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+      - name: Tag and release
+        run: |
+          git config user.name "GitHub Actions Bot"
+          git config user.email "<>"
+          version=v$(cat galaxy/Chart.yaml | grep ^version: | awk '{print $2}')
+          git tag -a $version -m "Automatic release of $version"
+          git push origin $version
+          gh release create $version --generate-notes --latest
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -5,6 +5,7 @@ on:
       - master
       - anvil
   pull_request: {}
+  workflow_dispatch: {}
 jobs:
   linting:
     runs-on: ubuntu-latest

diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-# Galaxy Helm Chart (v4)
+# Galaxy Helm Chart (v5)
 
 [Galaxy](https://galaxyproject.org/) is a data analysis platform focusing on
 accessibility, reproducibility, and transparency of primarily bioinformatics
@@ -8,7 +8,7 @@ updates, upgrades, and rollbacks.
 
 ## Supported software versions
 
-- Kubernetes 1.22+
+- Kubernetes 1.27+
 - Helm 3.5+
 
 ## Kubernetes cluster
@@ -34,18 +34,40 @@ helm upgrade --install ingress-nginx ingress-nginx \
 This chart relies on the features of other charts for common functionality:
 - [postgres-operator](https://github.com/zalando/postgres-operator) for the
   database;
-- [CSI-S3 chart](https://github.com/ctrox/csi-s3/pull/75/) for linking the
-  reference data to Galaxy and jobs based on S3FS.
+- [galaxy-cvmfs-csi](https://github.com/CloudVE/galaxy-cvmfs-csi-helm) for linking the
+  reference data to Galaxy and jobs based on CVMFS (default).
+- [csi-s3](https://github.com/ctrox/csi-s3/pull/75/) for linking
+  reference data to Galaxy and jobs based on S3FS (optional/alternative to CVMFS).
+- [rabbitmq-cluster-operator](https://github.com/rabbitmq/cluster-operator) for deploying
+  the message queue.
 
 In a production setting, especially if the intention is to run multiple Galaxies
 in a single cluster, we recommend installing the dependency charts separately
 once per cluster, and installing Galaxy with `--set postgresql.deploy=false
---set s3csi.deploy=false`.
+--set s3csi.deploy=false --set cvmfs.deploy=false --set rabbitmq.deploy=false`.
 
 ---
 
 ## Installing the chart
 
+### Using the chart from the packaged chart repo
+
+1. The chart is automatically packaged, versioned and uploaded to a helm repository
+on each accepted PR. Therefore, the latest version of the chart can be downloaded
+from this repository.
+
+```console
+helm repo add cloudve https://raw.githubusercontent.com/CloudVE/helm-charts/master/
+helm repo update
+```
+
+2. Install the chart with the release name `my-galaxy`. It is not advisable to
+   install Galaxy in the `default` namespace.
+
+```console
+helm install my-galaxy-release cloudve/galaxy
+```
+
 ### Using the chart from GitHub repo
 
 1. Clone this repository and add required dependency charts:
@@ -68,32 +90,50 @@ In several minute, Galaxy will be available at `/galaxy/` URL of your Kubernetes
 cluster. If you are running the development Kubernetes, Galaxy will be available
 at `http://localhost/galaxy/` (note the trailing slash).
 
-### Using the chart from the packaged chart repo
+## Uninstalling the chart
 
-1. Instead of using the source code repo, you can install the packaged version
-of the chart and hence not need to clone this GitHub repo. The packaged version
-may contain a bit older but possibly more stable code than what is the GitHub
-repo at any a given point in time.
+To uninstall/delete the `my-galaxy` deployment, run:
 
 ```console
-helm repo add cloudve https://raw.githubusercontent.com/CloudVE/helm-charts/master/
-helm repo update
+helm delete my-galaxy
 ```
 
-2. Install the chart with the release name `my-galaxy`. It is not advisable to
-   install Galaxy in the `default` namespace.
+if you see that some RabbitMQ and Postgres elements remain after some 10 minutes or more, you should be able to issue:
 
-```console
-helm install my-galaxy-release cloudve/galaxy
+```
+kubectl delete RabbitmqCluster/my-galaxy-rabbitmq-server
+kubectl delete statefulset/galaxy-my-galaxy-postgres
 ```
 
-## Uninstalling the chart
+it might be needed to remove the finalizer from the RabbitmqCluster if the above doesn't seem to get rid of RabbitmqCluster, through a
 
-To uninstall/delete the `my-galaxy` deployment, run:
+```
+kubectl edit RabbitmqCluster/my-galaxy-rabbitmq-server
+```
+
+remove the finalizer in:
+
+```
+apiVersion: rabbitmq.com/v1beta1
+kind: RabbitmqCluster
+metadata:
+  annotations:
+    meta.helm.sh/release-name: my-galaxy
+    meta.helm.sh/release-namespace: default
+  creationTimestamp: "2022-12-19T16:54:33Z"
+  deletionGracePeriodSeconds: 0
+  deletionTimestamp: "2022-12-19T17:41:40Z"
+  finalizers:
+  - deletion.finalizers.rabbitmqclusters.rabbitmq.com
+```
+
+and remove the postgres secret:
 
-```console
-helm delete my-galaxy
 ```
+kubectl delete secrets/standby.galaxy-my-galaxy-postgres.credentials.postgresql.acid.zalan.do
+```
+
+Consider as well that if you set persistence to be enabled, Postgres and Galaxy will leave their PVCs behind, which you might want to delete or not depending on your use case.
 
 ## Configuration
 
@@ -232,6 +272,8 @@ jobHandlers:
     failureThreshhold: 3
 ```
 
+# Additional Configurations
+
 ## Extra File Mappings
 
 The `extraFileMappings` field can be used to inject files to arbitrary paths in the `nginx` deployment, as well as any of the `job`, `web`, or `workflow` handlers, and the `init` jobs.
@@ -353,6 +395,9 @@ wildcard DNS mapping to `*.its.<host_name>`. To make Interactive Tools work on
 localhost, you can use `dnsmasq` or similar to handle wildcard DNS mappings for
 `*.localhost`.
 
+For linux:
+Follow the instructions here to configure dnsmasq on Linux: https://superuser.com/a/1718296
+
 For mac:
 ```bash
   $ brew install dnsmasq
@@ -380,28 +425,76 @@ The Galaxy application can be horizontally scaled for the web, job, or workflow
 by setting the desired values of the `webHandlers.replicaCount`,
 `jobHandlers.replicaCount`, and `workflowHandlers.replicaCount` configuration options.
 
-## Galaxy versions
+## Cron jobs
+
+Two Cron jobs are defined by default.  One to clean up Galaxy's database and one to clean up the `tmp` directory.  By default, these
+jobs run at 02:05 (the database maintenance script) and 02:15 (`tmp` directyory cleanup). Users can 
+change the times the cron jobs are run by changing the `schedule` field in the `values.yaml` file:
+
+```yaml
+cronJobs:
+  maintenance:
+    schedule: "30 6 * * *" # Execute the cron job at 6:30 UTC
+```
+or by specifying the `schedule` on the command line when instaling Galaxy:
+```bash
+# Schedule the maintenance job to run at 06:30 on the first day of each month
+helm install galaxy -n galaxy galaxy/galaxy --set cronJobs.maintenance.schedule="30 6 1 * *"
+```
+To disable a cron job after Galaxy has been deployed simply set the enabled flag for that job to false:
 
-Some changes introduced in the chart sometimes rely on changes in the Galaxy
-container image, especially in relation to the Kubernetes runner. This table
-keeps track of recommended Chart versions for particular Galaxy versions as
-breaking changes are introduced. Otherwise, the Galaxy image and chart should be
-independently upgrade-able. In other words, upgrading the Galaxy image from
-`21.05` to `21.09` should be a matter of `helm upgrade my-galaxy cloudve/galaxy
---reuse-values --set image.tag=21.09`.
 
+```bash
+helm upgrade galaxy -n galaxy galaxy/galaxy --reuse-values --set cronJobs.maintenance.enabled=false
+```
+
+### Run a CronJob manually
+
+Cron jobs can be invoked manually with tools such as [OpenLens](https://github.com/MuhammedKalkan/OpenLens)
+or from the command line with `kubectl`
+```bash
+kubectl create job --namespace <namespace> <job name> --from cronjob/galaxy-cron-maintenance 
+```
+This will run the cron job regardless of the `schedule` that has been set.
+
+**Note:** the name of the cron job will be `{{ .Release.Name }}-cron-<job name>` where the `<job name>`
+is the name (key) used in the `values.yaml` file.
+
+### CronJob configuration
+
+The following fields can be specified when defining cron jobs.
+
+| Name | Definition                                                                                                                                | Required |
+|---|-------------------------------------------------------------------------------------------------------------------------------------------|----------|
+| enabled | `true` or `false`.  If `false` the cron job will not be run.  Default is `true` | **Yes**  |
+| schedule | When the job will be run.  Use tools such as [crontab.guru](https://crontab.guru) for assistance determining the proper schedule string   | **Yes**  |
+| defaultEnv | `true` or `false`. See the `galaxy.podEnvVars` macro in `_helpers.tpl` for the list of variables that will be defined. Default is `false` | No       |
+| extraEnv | Define extra environment variables that will be available to the job | No       |
+| securityContext | Specifies a `securityContext` for the job. Typically used to set `runAsUser` | No       |
+| image | Specify the Docker container used to run the job | No       |
+| command | The command to run | **Yes**  |
+| args | Any command line arguments that should be passed to the `command` | No       |
+| extraFileMappings | Allow arbitrary files to be mounted from config maps | No       |
+
+### Notes
+
+If specifying the Docker `image` both the `resposity` and `tag` MUST be specified.
+```yaml
+  image:
+    repository: quay.io/my-organization/my-image
+    tag: "1.0"  
+```
+
+The `extraFileMappings` block is similar to the global `extraFileMappings` except the file will only be mounted for that cron job.
+The following fields can be specified for each file.
 
-| Chart version        | Galaxy version   | Description     |
-| :------------------ | :--------------- | :-------------- |
-| `5.0`               | `22.05`          | Needs at least container image 22.05 as Galaxy switched from uwsgi to gunicorn  |
-| `4.0`               | `21.05`          | Needs [Galaxy PR#11899](https://github.com/galaxyproject/galaxy/pull/11899) for eliminating the CVMFS. If running chart 4.0+ with Galaxy image `21.01` or below, use the CVMFS instead with `--set setupJob.downloadToolConfs.enabled=false --set cvmfs.repositories.cvmfs-gxy-cloud=cloud.galaxyproject.org --set cvmfs.galaxyPersistentVolumeClaims.cloud.storage=1Gi --set cvmfs.galaxyPersistentVolumeClaims.cloud.storageClassName=cvmfs-gxy-cloud --set cvmfs.galaxyPersistentVolumeClaims.cloud.mountPath=/cvmfs/cloud.galaxyproject.org` |
+| Name | Definition | Required |
+|---|---|----------|
+| mode | The file mode (permissions) assigned to the file | No       |
+| tpl | If set to `true` the file contents will be run through Helm's templating engine. Defaults to `false` | No       |
+| content | The contents of the file | **Yes**  | 
 
-## Funding
 
-- _Version 3+_: Galaxy Project, Genomics Virtual Laboratory (GVL)
+See the `example` cron job included in the `values.yaml` file for a full example.
 
-- _Version 2_: Genomics Virtual Laboratory (GVL), Galaxy Project, and European
-  Commission (EC) H2020 Project PhenoMeNal, grant agreement number 654241.
 
-- _Version 1_: European Commission (EC) H2020 Project PhenoMeNal, grant
-  agreement number 654241.
diff --git a/VALUES.md b/VALUES.md
@@ -0,0 +1,50 @@
+| Key | Description |
+|-----|-------------|
+| nameOverride | Partial override of the `galaxy.fullname`.  The `.Release.Name` will be prepended to generate the fullname. |
+| fullnameOverride | Fully override the `galaxy.fullname` |
+| image.repository | Repository containing the Galaxy image. |
+| image.tag | Galaxy Docker image tag (generally corresponds to the desired Galaxy version) |
+| image.pullPolicy | Galaxy image [pull policy](https://kubernetes.io/docs/concepts/configuration/overview/#container-images) |
+| imagePullSecrets | Secrets used to [access a Galaxy image from a private repository](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) |
+| trainingHook.enabled | Enable the GTN webhook to link references to tools in tutorials to the corresponding tool panel in Galaxy. |
+| trainingHook.url | The training material server used to service the training-material webhook. |
+| service.type | The Galaxy service type |
+| service.port | The port Galaxy is listening to |
+| service.nodePort | The external port exposed on each node |
+| metrics.enabled | Enable the metrics server. Defaults to `false` |
+| serviceAccount.create | Specifies whether a service account should be created |
+| serviceAccount.annotations | Annotations to add to the service account |
+| serviceAccount.name | The name of the service account to use. If not set and create is true, a name is generated using the fullname template |
+| rbac.enabled | Does the cluster use role based access control. |
+| securityContext.fsGroup | Security context and file system group used by jobs. |
+| persistence | Configure the PVC used by Galaxy for local storage. |
+| persistence.enabled | Persistence is enabled by default |
+| persistence.name | Name of the PVC to create |
+| persistence.storageClass | StorageClass for the PVC. Must support `ReadWriteMany`. |
+| persistence.existingClaim | The name of an existing PVC to use for persistence. |
+| setupJob | tasks to perform once after installation |
+| setupJob.createDatabase | create the database |
+| setupJob.securityContext.runAsUser | the setup jobs will run as this user |
+| setupJob.securityContext.runAsGroup | the `runAsUser` will belong to this group. |
+| setupJob.securityContext.fsGroup | the filesystem group |
+| setupJob.downloadToolConfs.archives.startup | A tar.gz publicly accessible archive containing AT LEAST conf files and XML tool wrappers. Meant to be enough for Galaxy handlers to startup |
+| setupJob.downloadToolConfs.archives.running | A tar.gz publicly accessible archive containing AT LEAST confs, tool wrappers, and scripts excluding test data. Meant to be enough for Galaxy handlers to run jobs. |
+| setupJob.downloadToolConfs.archives.full | A tar.gz publicly accessible archive containing the full `tools` directory, including each tool's test data. Meant to be enough to run automated tool-tests, fully mimicking CVMFS setup |
+| extraInitContainers | Allow users to specify extra init containers |
+| ingress.enabled | Should ingress be enabled. Defaults to `true` |
+| ingress.ingressClassName |  |
+| resources.requests | We recommend updating these based on the usage levels of the server. |
+| postgresql.deploy | Whether to deploy the postgresl operator. In general, we recommend installing the operator globally in production. |
+| postgresql.existingDatabase | hostname and port of an existing database to use. |
+| refdata | Configuration block for reference data |
+| refdata.enabled | Whether or not to mount cloud-hosted Galaxy reference data and tools. |
+| refdata.type | `s3fs` or `cvmfs`, determines the CSI to use for mounting reference data. `cvmfs` is the default and recommended for the time being. |
+| cvmfs | Configuration block if `cvmfs` is used as `refdata.type` |
+| cvmfs.deploy | Deploy the Galaxy-CVMFS-CSI Helm Chart. This is an optional dependency, and for production scenarios it should be deployed separately as a cluster-wide resource |
+| s3csi | Configuration block if `s3csi` is used as the `refdata.type` |
+| s3csi.deploy | Deploy the CSI-S3 Helm Chart. This is an optional dependency, and for production scenarios it should be deployed separately as a cluster-wide resource. |
+| useSecretConfigs | When this flag is set to true, all configs will be set in secrets, when it is set to false, all configs will be set in configmaps |
+| configs | All config files will be relative to `/galaxy/server/config/` directory |
+| configs.galaxy\.yml | Galaxy configuration. See the [Galaxy documentation](https://docs.galaxyproject.org/en/master/admin/config.html) for more information. |
+| jobs | Additional dynamic rules to map into the container. |
+| jobs.priorityClass.enabled | Assign a [priorityClass](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass) to the dispatched jobs. |
diff --git a/galaxy/Chart.yaml b/galaxy/Chart.yaml
@@ -1,8 +1,8 @@
 apiVersion: v2
 name: galaxy
 type: application
-version: 5.5.0
-appVersion: "22.05"
+version: 5.14.2
+appVersion: "24.0.2"
 description: Chart for Galaxy, an open, web-based platform for accessible, reproducible, and transparent computational biomedical research.
 icon: https://galaxyproject.org/images/galaxy-logos/galaxy_project_logo_square.png
 dependencies:
@@ -15,14 +15,14 @@ dependencies:
       - deploy-postgres
   - name: csi-s3
     repository: https://raw.githubusercontent.com/cloudve/helm-charts/master/
-    version: 0.31.3
+    version: 0.35.2
     condition: s3csi.deploy
     alias: s3csi
     tags:
       - deploy-s3csi
   - name: galaxy-cvmfs-csi
     repository: https://raw.githubusercontent.com/cloudve/helm-charts/master/
-    version: 2.0.0
+    version: 2.4.0
     condition: cvmfs.deploy
     alias: cvmfs
     tags:

diff --git a/galaxy/disabled/configmap-galaxy.yaml b/galaxy/disabled/configmap-galaxy.yaml
@@ -0,0 +1,9 @@
+apiVersion: v1
+metadata:
+  name: {{ .Release.Name }}-galaxy-config
+  labels:
+    {{- include "galaxy.labels" $ | nindent 4 }}
+kind: ConfigMap
+data:
+  galaxy.yml: |
+    {{- .Values.galaxy | toYaml | nindent 4 }}
diff --git a/galaxy/files/configs/tool_conf.xml b/galaxy/files/configs/tool_conf.xml
@@ -38,6 +38,10 @@
     <tool file="${model_tools_path}/build_list.xml" />
     <tool file="${model_tools_path}/extract_dataset.xml" />
   </section>
+  <section id="expression_tools" name="Expression Tools">
+    <tool file="expression_tools/parse_values_from_file.xml"/>
+    <tool file="expression_tools/pick_value.xml"/>
+  </section>
   <label id="general_text_label" text="General Text Tools" />
   <section id="text_manipulation" name="Text Manipulation">
     <tool file="filters/fixedValueColumn.xml" />