Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cron job maintenance #448

Merged
merged 30 commits into from
Apr 18, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4138130
Remove spaces that makes helm lint unhappy
ksuderman Feb 9, 2024
e0b960d
First pass at defining cron jobs in the values.yaml file.
ksuderman Feb 9, 2024
9c84d17
Fix command formatting and quoting
ksuderman Feb 13, 2024
d2941d7
Remove nodeSelector
ksuderman Feb 13, 2024
95474e2
Tmp cleanup and maintenance scripts are treated as special cases.
ksuderman Feb 16, 2024
93930d7
Update the name of the container used to run the maintenance script.
ksuderman Feb 16, 2024
287cf13
Define all cron jobs in the values.yaml file again, but allows time d…
ksuderman Feb 19, 2024
8225be5
Use walltime limit for cleanup and other minor tweaks
nuwang Feb 20, 2024
8cc5d94
Change find units from seconds to days
nuwang Feb 20, 2024
64320c8
Parameterize the nodeSelector
ksuderman Feb 21, 2024
2bbaa3a
Remove the chown cron job
ksuderman Feb 21, 2024
df1fb39
Reverting my revert
ksuderman Feb 21, 2024
247cbf5
Add extraFileMappings for cron jobs
ksuderman Feb 28, 2024
cf77963
Start documenting cron jobs
ksuderman Feb 28, 2024
13e4f0b
Make the galaxy.yml file available in a configmap for the maintenance…
ksuderman Mar 8, 2024
27fd304
Update galaxy/values.yaml
ksuderman Mar 8, 2024
42488ff
Add helper to calculate the postgres connection string
ksuderman Mar 10, 2024
0fafea8
Add cron job documentation and remove Galaxy versions section
ksuderman Mar 10, 2024
497b572
Add env definitions
ksuderman Mar 12, 2024
da6a97a
Maintenace job should include default env vars
ksuderman Mar 12, 2024
cdc2a98
Update galaxy/values.yaml
ksuderman Mar 12, 2024
63bc997
Update galaxy/values.yaml
ksuderman Mar 12, 2024
d96979a
Parameterize Docker image for cron jobs
ksuderman Mar 12, 2024
3af24cd
Allow mode (permissions) to be defined on extraFileMappings
ksuderman Mar 13, 2024
3fe5723
Add example cron job
ksuderman Mar 13, 2024
07fdbd4
Additional documentation for cron jobs
ksuderman Mar 13, 2024
b4cbdf6
Comment out the example cron job. Examples should not add arbitrary c…
ksuderman Mar 14, 2024
db82ef0
Run extraEnv values through the template engine
ksuderman Mar 17, 2024
a7c1aed
Merge branch 'master' into 408-maintenance
ksuderman Apr 18, 2024
14f190d
Add newline to the end of the file.
ksuderman Apr 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 26 additions & 24 deletions galaxy/templates/cronjob-maintenance.yaml
Original file line number Diff line number Diff line change
@@ -1,47 +1,49 @@
{{ range $key, $cronjob := .Values.cronJobs }}
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "galaxy.fullname" . }}-maintenance
name: {{ include "galaxy.fullname" $ }}-maintenance-{{ $key }}
labels:
{{- include "galaxy.labels" . | nindent 4 }}
{{- include "galaxy.labels" $ | nindent 4 }}
spec:
schedule: "0 2 * * *"
schedule: {{ $cronjob.schedule | quote }}
jobTemplate:
spec:
template:
spec:
{{- if $cronjob.securityContext }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
{{- with .Values.nodeSelector }}
{{- toYaml $cronjob.securityContext | nindent 12 }}
{{- end}}
{{- with $.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 16 }}
{{- toYaml . | nindent 12 }}
{{- end }}
containers:
- name: galaxy-maintenance
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
# delete all tmp files older than walltime limit
- name: galaxy-maintenance-{{ $key }}
image: "{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
imagePullPolicy: {{ $.Values.image.pullPolicy }}
command:
- find
- {{ .Values.persistence.mountPath }}/tmp
- '!'
- -newermt
- -{{ (index .Values "configs" "job_conf.yml" "runners" "k8s" "k8s_walltime_limit" | default 604800) }} seconds
- -type
- f
- -exec
- rm
- '{}'
- ;
{{- range $cmd := $cronjob.command }}
- {{ tpl $cmd $ | quote }}
{{- end}}
{{- if $cronjob.args }}
args:
{{- range $arg := $cronjob.args }}
- {{ tpl $arg $ | quote }}
{{- end }}
{{- end }}
volumeMounts:
- name: galaxy-data
mountPath: {{ .Values.persistence.mountPath }}
mountPath: {{ $.Values.persistence.mountPath }}
volumes:
- name: galaxy-data
{{- if .Values.persistence.enabled }}
{{- if $.Values.persistence.enabled }}
persistentVolumeClaim:
claimName: {{ template "galaxy.pvcname" . }}
claimName: {{ template "galaxy.pvcname" $ }}
{{- else }}
emptyDir: {}
{{- end }}
restartPolicy: OnFailure
{{- end }}
2 changes: 1 addition & 1 deletion galaxy/templates/hook-cvmfs-fix.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{{- if and .Values.refdata.enabled (eq .Values.refdata.type "cvmfs") }}
# Include the code you want to run when both conditions are met
# Include the code you want to run when both conditions are met
apiVersion: batch/v1
kind: Job
metadata:
Expand Down
47 changes: 47 additions & 0 deletions galaxy/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,53 @@ extraEnv: []
# - name: EXAMPLE_ENV
# value: MY_VALUE

#- User defined CronJobs
cronJobs:
chown:
schedule: "0 * * * *"
securityContext:
runAsUser: 0
command:
- /usr/bin/chown
args:
- "-R"
- "galaxy"
- "/galaxy/server/database"
ksuderman marked this conversation as resolved.
Show resolved Hide resolved
maintenance:
schedule: "5 2 * * *"
extraSettings:
days: 7
securityContext:
runAsUser: 0
command:
- "/bin/bash"
- "-c"
args:
- "/galaxy/server/scripts/maintenance.sh"
- "--no-dry-run"
- "--days"
- "{{ .Values.cronJobs.maintenance.extraSettings.days }}"
#- Remove files from the tmp directory that are older than the allowable wall time for a job
tmp:
schedule: "15 2 * * *"
extraSettings:
seconds: 604800
ksuderman marked this conversation as resolved.
Show resolved Hide resolved
securityContext:
runAsUser: 0
command:
- /usr/bin/find
args:
- "{{ .Values.persistence.mountPath }}/tmp"
- "!"
- "-newermt"
- "{{ .Values.cronJobs.tmp.extraSettings.seconds }}"
ksuderman marked this conversation as resolved.
Show resolved Hide resolved
- "seconds"
- "-type"
- "f"
- "-exec"
- "rm"
- "{}"
- ";"
ingress:
#- Should ingress be enabled. Defaults to `true`
enabled: true
Expand Down
Loading