Skip to content

Commit 8af2745

Browse files
HenrrypgHenrrypg
authored
feat: add auto generated jobs (#87)
1 parent cb8bcb2 commit 8af2745

23 files changed

+471
-588
lines changed

‎README.md

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
Drydock
2+
=======
3+
4+
Drydock is an opinionated tool offering a set of Tutor plugins aiming to provide features that enhance the operation of OpenedX installations in Kubernetes. It is developed by [Edunext](https://www.edunext.co/).
5+
6+
- A set of Kubernetes Jobs that replace the current tutor jobs with [ArgoCD Sync Waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) that allow for a more controlled deployment of openedx.
7+
- A set of Kustomization overrides adding [ArgoCD Sync Waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) annotations to needed additional resources such as debug, workers or hpa.
8+
- Backup cronjobs that allow backup of the MySQL and MongoDB databases.
9+
- Integration of New Relic monitoring
10+
- Add flower deployment for Celery
11+
- Add a custom nginx and cert-manager configuration
12+
- Add a set of debug resources to help diagnose issues
13+
14+
Extra plugins added:
15+
16+
- A patch that allows for the tuning of celery workers via `DRYDOCK_ENABLE_CELERY_TUNING`
17+
- Allows caddy to catch requests for multiple domains through `DRYDOCK_ENABLE_MULTITENANCY`
18+
- Add scorm matcher to caddy through `DRYDOCK_ENABLE_SCORM`
19+
- A patch that allows for the use of sentry via `DRYDOCK_ENABLE_SENTRY` and `DRYDOCK_SENTRY_DSN`
20+
- Patch for cms and lms worker pods to allow pod probes and lifecycle to work properly. Enabled via `DRYDOCK_POD_LIFECYCLE`
21+
22+
Installation
23+
------------
24+
25+
``` bash
26+
tvm plugins install -e git+https://github.com/edunext/drydock#egg=drydock
27+
tutor plugins enable drydock
28+
tutor config save
29+
```
30+
31+
Getting started
32+
---------------
33+
34+
``` bash
35+
tutor config save
36+
```
37+
38+
Configuration
39+
-------------
40+
41+
The following configuration options are available:
42+
43+
- `DRYDOCK_INIT_JOBS`: Whether run the initialization jobs or not. Defaults to `false`.
44+
- `DRYDOCK_CMS_SSO_USER`: The username of the CMS SSO user. Defaults to `cms`.
45+
- `DRYDOCK_AUTO_TLS`: Whether to use cert-manager to automatically generate TLS certificates. Defaults to `false`.
46+
- `DRYDOCK_FLOWER`: Whether to deploy a flower deployment for celery. Defaults to `false`.
47+
- `DRYDOCK_INGRESS`: Whether to deploy an ingress for the LMS and CMS. Defaults to `false`.
48+
- `DRYDOCK_INGRESS_EXTRA_HOSTS`: A list of extra hosts to add to the ingress. Defaults to `[]`.
49+
- `DRYDOCK_CUSTOM_CERTS`: A dictionary of custom certificates to use with cert-manager. Defaults to `{}`.
50+
- `DRYDOCK_NEWRELIC_LICENSE_KEY`: The New Relic license key. Defaults to `""`.
51+
- `DRYDOCK_DEBUG`: Whether to deploy debug resources. Defaults to `false`.
52+
- `DRYDOCK_ENABLE_CELERY_TUNING`: Whether to enable celery tuning. Defaults to `true`.
53+
- `DRYDOCK_ENABLE_MULTITENANCY`: Whether to enable multitennacy. Defaults to `true`.
54+
- `DRYDOCK_ENABLE_SCORM`: Whether to enable scorm. Defaults to `true`.
55+
- `DRYDOCK_ENABLE_SENTRY`: Whether to enable sentry. Defaults to `true`.
56+
- `DRYDOCK_SENTRY_DSN`: The sentry DSN. Defaults to `""`.
57+
- `DRYDOCK_POD_LIFECYCLE`: Whether to enable pod lifecycle. Defaults to `true`.
58+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_MFE`: The minimum available percentage for the MFE's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
59+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_FORUM`: The minimum available percentage for the FORUM's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
60+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CADDY`: The minimum available percentage for the CADDY's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
61+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_LMS`: The minimum available percentage for the LMS's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
62+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_LMS_WORKER`: The minimum available percentage for the LMS WORKER's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
63+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CMS`: The minimum available percentage for the CMS's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
64+
- `DRYDOCK_PDB_MINAVAILABLE_PERCENTAGE_CMS_WORKER`: The minimum available percentage for the worker's PodDisruptionBudget. To disable the PodDisruptionBudget, set `0`. Defaults to `0`.
65+
- `DRYDOCK_MIGRATE_FROM`: it allows defining the version of the OpenedX platform we are migrating from. It accepts the integer value mapping the origin release, for instance, `13`(maple) or `14`(nutmeg). When this variable is set, a group of `release-specific upgrade jobs` are added to the Kubernetes manifests. These jobs are applied to the cluster in a suitable order (thanks to the GitOps implementation with ArgoCD + sync waves) to guarantee the correct behavior of the platform in the new version. This brings the `tutor k8s upgrade <https://github.com/overhangio/tutor/blob/v15.3.7/tutor/commands/k8s.py#L484>`_ command to the GitOps pattern. The release-specific upgrade jobs are supported from release `13`(maple). Defaults to `0` (which disables release-specific upgrade jobs)
66+
67+
> **_NOTE:_** You also need to set `DRYDOCK_INIT_JOBS` to `true` to enable the release-specific upgrade jobs in the case of a platform migration.
68+
69+
Job generation
70+
--------------
71+
72+
Tutor doesn't generate manifest files for the initialization jobs, in consequence we can't use GitOps tools like ArgoCD to deploy the initialization jobs.
73+
74+
We had been using a static definition of the initialization jobs, but now we are using the `Tutor filters <https://docs.tutor.edly.io/reference/api/hooks/filters.html>`_ to generate the kubernetes definition of the initialization jobs. This is a big improvement because now we can add new initialization jobs without modifying the Drydock code. The jobs are taken from `COMMANDS_PRE_INIT`, `COMMANDS_INIT` and `CLI_DO_INIT_TASKS` Filters.
75+
76+
ArgoCD Sync Waves Support
77+
-----------------------
78+
79+
[Tutor filter](https://docs.tutor.edly.io/reference/api/hooks/filters.html) **SYNC_WAVES_ORDER** was added to allow define [ArgoCD Sync Waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) order and apply to the kubernetes resources through **get_sync_waves_for_resource** function.
80+
81+
We are defined by defult the following order:
82+
83+
1. `All kubernetes resources` (except the ones that are defined in the next waves)
84+
2. `Initialization Jobs`
85+
3. `Upgrade Jobs`: When **DRYDOCK_MIGRATE_FROM** is set, over the Sync Wave 50
86+
4. `CMS and LMS Deployments`: When **DRYDOCK_POD_LIFECYCLE** is active, over the Sync Wave 100
87+
5. `Debug Resources`: When **DRYDOCK_DEBUG** active, over the Sync Wave 100
88+
6. `Horizontal Pod Autoscalers`: When active, over the Sync Wave 150
89+
90+
Workaround to upgrade from Maple to Palm or later
91+
-------------------------------------------------
92+
93+
> **_NOTE:_** Quince uses Django 4.2 which only supports MySQL 8 or higher. You must upgrade your version of MySQL prior to performing the upgrade.
94+
95+
The upgrade from Maple to Palm fails because an issue with a squashed migration in [edx-enterprise](https://github.com/openedx/edx-enterprise/blob/3.61.11/integrated_channels/blackboard/migrations/0001_initial_squashed_0014_alter_blackboardlearnerassessmentdatatransmissionaudit_enterprise_course_enrollment_id.py). To go around this issue, we need to apply migrations using an older version of edx-enterprise (3.60.4).
96+
97+
1. Run the sync to Palm without enabling the init jobs or upgrade jobs.
98+
99+
2. Once the LMS Deployment is running in the Palm version, go inside a pod and run the following:
100+
101+
``` bash
102+
pip install edx-enterprise==3.60.4
103+
./manage.py lms migrate
104+
pip install edx-enterprise==3.61.11
105+
```
106+
107+
3. Now, you can enable the init jobs and upgrade jobs and run the sync again.
108+
109+
This workaround references the [Andrey's comment](https://discuss.openedx.org/t/updating-tutor-lilac-to-palm-now-that-palms-released-fails/10557/23)
110+
111+
Rationale
112+
---------
113+
114+
This project is proposed as a possible way of creating a community maintained
115+
reference for large openedx installation.
116+
Sometimes the needs for customization in large instances of openedx goes
117+
against the required simplicity in the configuration that the tutor project
118+
strives for. In those cases, the solution is to create a tutor plugin that
119+
allows for such advanced customization options.
120+
121+
This projects intends to fill that gap with a solution that should allow many
122+
community members to collaborate in one repo on the heavy toll that is the
123+
maintainance of openedx operations.
124+
125+
License
126+
-------
127+
128+
This software is licensed under the terms of the AGPLv3.

‎README.rst

Lines changed: 0 additions & 85 deletions
This file was deleted.

‎drydock/hooks.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
"""
2+
These hooks are stored in a separate module. If they were included in plugin.py, then
3+
the pod-autoscaling hooks would be created in the context of some other plugin that imports
4+
them.
5+
"""
6+
7+
from __future__ import annotations
8+
9+
import typing as t
10+
11+
from tutor.core.hooks import Filter
12+
13+
SYNC_WAVES_ORDER_ATTRS_TYPE = t.Dict[str, int]
14+
15+
SYNC_WAVES_ORDER: Filter[dict[str, SYNC_WAVES_ORDER_ATTRS_TYPE], []] = Filter()

‎drydock/patches/k8s-jobs

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: mongodb-job
6+
labels:
7+
app.kubernetes.io/component: job
8+
spec:
9+
ttlSecondsAfterFinished: 100
10+
template:
11+
spec:
12+
restartPolicy: Never
13+
containers:
14+
- name: mongodb
15+
image: {{ DOCKER_IMAGE_MONGODB }}

‎drydock/patches/kustomization

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,21 +13,14 @@ patches:
1313
kind: Job
1414
labelSelector: app.kubernetes.io/component=job
1515
path: plugins/drydock/k8s/patches/tutor-jobs.yml
16-
# Patch the sync waves
17-
- target:
18-
kind: Deployment
19-
name: "lms|cms|lms-worker|cms-worker|forum"
20-
path: plugins/drydock/k8s/patches/sync-wave-4.yml
21-
{%- if DRYDOCK_DEBUG is defined and DRYDOCK_DEBUG %}
22-
- target:
23-
kind: Deployment|Ingress|Service
24-
name: "cms-debug|lms-debug|ingress-debug"
25-
path: plugins/drydock/k8s/patches/sync-wave-5.yml
26-
{%- endif %}
27-
- target:
28-
kind: HorizontalPodAutoscaler
29-
path: plugins/drydock/k8s/patches/sync-wave-5.yml
3016
{% if DRYDOCK_ENABLE_CELERY_TUNING %}
3117
- path: plugins/drydock/k8s/celery/cms-worker.yml
3218
- path: plugins/drydock/k8s/celery/lms-worker.yml
3319
{% endif -%}
20+
- target:
21+
kind: HorizontalPodAutoscaler
22+
path: plugins/drydock/k8s/patches/hpa-sync-wave.yml
23+
- target:
24+
kind: Deployment
25+
name: '{% for name in DRYDOCK_POST_INIT_DEPLOYMENTS %}{{ name }}{% if not loop.last %}|{% endif %}{% endfor %}'
26+
path: plugins/drydock/k8s/patches/post-init-deployments-sync-wave.yml

‎drydock/patches/kustomization-resources

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,7 @@
11
- plugins/drydock/k8s/multipurpose-jobs.yml
22
- plugins/drydock/k8s/pdb/openedx-pdb.yml
33
{%- if DRYDOCK_INIT_JOBS %}
4-
- plugins/drydock/k8s/drydock-jobs/mysql.yml
5-
- plugins/drydock/k8s/drydock-jobs/mongodb.yml
6-
- plugins/drydock/k8s/drydock-jobs/lms.yml
7-
- plugins/drydock/k8s/drydock-jobs/cms.yml
8-
- plugins/drydock/k8s/drydock-jobs/minio.yml
9-
- plugins/drydock/k8s/drydock-jobs/forum.yml
10-
- plugins/drydock/k8s/drydock-jobs/notes.yml
4+
- plugins/drydock/k8s/jobs.yml
115
{%- endif %}
126
{% if DRYDOCK_FLOWER -%}
137
- plugins/drydock/k8s/flower.yml

0 commit comments

Comments
 (0)