Skip to content

Commit

Permalink
valkey tests
Browse files Browse the repository at this point in the history
  • Loading branch information
cunla committed Sep 14, 2024
1 parent bf5baa4 commit 902944d
Show file tree
Hide file tree
Showing 5 changed files with 52 additions and 39 deletions.
33 changes: 31 additions & 2 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,39 @@ If no queues are specified, will run on default queue only.
All queues must have the same redis settings on `SCHEDULER_QUEUES`.

```shell
python manage.py rqworker queue1 queue2 queue3

usage: manage.py rqworker [-h] [--pid PIDFILE] [--burst] [--name NAME] [--worker-ttl WORKER_TTL] [--max-jobs MAX_JOBS] [--fork-job-execution FORK_JOB_EXECUTION]
[--job-class JOB_CLASS] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color]
[--skip-checks]
[queues ...]

positional arguments:
queues The queues to work on, separated by space, all queues should be using the same redis

options:
--pid PIDFILE file to write the worker`s pid into
--burst Run worker in burst mode
--name NAME Name of the worker
--worker-ttl WORKER_TTL
Default worker timeout to be used
--max-jobs MAX_JOBS Maximum number of jobs to execute before terminating worker
--fork-job-execution FORK_JOB_EXECUTION
Fork job execution to another process
--job-class JOB_CLASS
Jobs class to use
--version Show program's version number and exit.
-v {0,1,2,3}, --verbosity {0,1,2,3}
Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output
--settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used.
--pythonpath PYTHONPATH
A directory to add to the Python path, e.g. "/home/djangoprojects/myproject".
--traceback Raise on CommandError exceptions.
--no-color Don't colorize the command output.
--force-color Force colorization of the command output.
--skip-checks Skip system checks.
```
## export
Export all scheduled tasks from django db to json/yaml format.
Expand Down
1 change: 1 addition & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@
'DEFAULT_RESULT_TTL': 500,
'DEFAULT_TIMEOUT': 300, # 5 minutes
'SCHEDULER_INTERVAL': 10, # 10 seconds
'BROKER': 'redis', #
}
```

Expand Down
52 changes: 18 additions & 34 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,15 @@ You can set in `settings.py` a default value for `DEFAULT_RESULT_TTL` and `DEFAU

```python
# settings.py
RQ = {
SCHEDULER_CONFIG = {
'DEFAULT_RESULT_TTL': 360,
'DEFAULT_TIMEOUT': 60,
}
```

## Scheduling a job Through django-admin

* Sign in to the Django Admin site (e.g., http://localhost:8000/admin/) and locate the
**Tasks Scheduler** section.
* Sign in to the Django Admin site (e.g., http://localhost:8000/admin/) and locate the `Tasks Scheduler` section.
* Click on the **Add** link for the type of job you want to add (`Scheduled Task` - run once, `Repeatable Task` - run
multiple times, `Cron Task` - Run based on cron schedule).
* Enter a unique name for the job in the **Name** field.
Expand Down Expand Up @@ -83,49 +82,28 @@ calculated in runtime.

![](media/add-args.jpg)

### Scheduled Task - run once
### Scheduled Task: run once

No additional steps required.
No additional steps are required.

### Repeatable Task - Run a job multiple time based on interval
### Repeatable Task: Run a job multiple time based on interval

Additional fields required:
These additional fields are required:

* Enter an **Interval**, and choose the **Interval unit**. This will calculate the time before the function is called
again.
* In the **Repeat** field, enter the number of time the job is to be run. Leaving the field empty, means the job will
be scheduled to run forever.

### Cron Task - Run a job multiple time based on cron
### Cron Task: Run a job multiple times based on cron

Additional fields required:
These additional fields are required:

* In the **Repeat** field, enter the number of time the job is to be run. Leaving the field empty, means the job will be
scheduled to run forever.
* In the **cron string** field, enter a cron string describing how often the job should run.

### Scheduled Task - run once

No additional steps required.

### Repeatable Task - Run a job multiple time based on interval

Additional fields required:

* Enter an **Interval**, and choose the **Interval unit**. This will calculate the time before the function is called
again.
* In the **Repeat** field, enter the number of time the job is to be run. Leaving the field empty, means the job will
be scheduled to run forever.

### Cron Task - Run a job multiple time based on cron

Additional fields required:

* In the **Repeat** field, enter the number of time the job is to be run. Leaving the field empty, means the job will be
scheduled to run forever.
* In the **cron string** field, enter a cron string describing how often the job should run.

## Enqueue jobs through command line
## Enqueue jobs using the command line

It is possible to queue a job to be executed from the command line
using django management command:
Expand All @@ -134,18 +112,24 @@ using django management command:
python manage.py run_job -q {queue} -t {timeout} -r {result_ttl} {callable} {args}
```

## Running a worker
## Running a worker to process queued jobs in the background

Create a worker to execute queued jobs on specific queues using:

```shell
python manage.py rqworker [queues ...]
python manage.py rqworker [-h] [--pid PIDFILE] [--burst] [--name NAME] [--worker-ttl WORKER_TTL] [--max-jobs MAX_JOBS] [--fork-job-execution FORK_JOB_EXECUTION]
[--job-class JOB_CLASS] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color]
[--skip-checks]
[queues ...]

```

More information about the different parameters can be found in the [commands documentation](commands.md).

### Running multiple workers as unix/linux services using systemd

You can have multiple workers running as system services.
In order to have multiple rqworkers, edit the `/etc/systemd/system/rqworker@.service`
To have multiple rqworkers, edit the `/etc/systemd/system/rqworker@.service`
file, make sure it ends with `@.service`, the following is example:

```ini
Expand Down
3 changes: 1 addition & 2 deletions scheduler/admin/task_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
from django.utils.translation import gettext_lazy as _

from scheduler import tools
from scheduler.connection_types import ConnectionErrorType
from scheduler.models import CronTask, TaskArg, TaskKwarg, RepeatableTask, ScheduledTask
from scheduler.settings import SCHEDULER_CONFIG, logger
from scheduler.tools import get_job_executions
Expand Down Expand Up @@ -187,7 +186,7 @@ def change_view(self, request, object_id, form_url="", extra_context=None):
obj = self.get_object(request, object_id)
try:
execution_list = get_job_executions(obj.queue, obj)
except (redis.ConnectionError, valkey.ConnectionError) as e:
except (redis.ConnectionError, valkey.ConnectionError) as e:
logger.warn(f"Could not get job executions: {e}")
execution_list = list()
paginator = self.get_paginator(request, execution_list, SCHEDULER_CONFIG.EXECUTIONS_IN_PAGE)
Expand Down
2 changes: 1 addition & 1 deletion scheduler/connection_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
PipelineType = Union[redis.client.Pipeline, valkey.client.Pipeline]
RedisSentinel = redis.sentinel.Sentinel

BrokerConnectionClass: Dict[Tuple[Broker,bool], Type] = {
BrokerConnectionClass: Dict[Tuple[Broker, bool], Type] = {
# Map of (Broker, Strict flag) => Connection Class
(Broker.REDIS, False): redis.Redis,
(Broker.VALKEY, False): valkey.Valkey,
Expand Down

0 comments on commit 902944d

Please sign in to comment.