Skip to content

Redis backend: apply Postgres scale improvements (scheduling, activity type filtering) #72

@alob-mtc

Description

@alob-mtc

Follow-up from #70 (addressed for PostgreSQL in #71). The Redis backend still has the original limitations:

  1. process_scheduled() runs on every node — the Lua script moves at most 100 items per poll. Every WorkerEngine instance runs this loop independently, creating redundant Redis traffic that grows with the number of nodes.

  2. dequeue() ignores activity_types filter — the signature accepts the parameter but the value is discarded. There is no workload isolation for the Redis backend yet.

  3. No key partitioning — a single sorted set per queue_name for both the main queue and the scheduled set.

What needs to happen

  • Apply native scheduling in Redis dequeue() (or coordinate process_scheduled() to run on a single node)
  • Implement activity_types filtering in the Redis dequeue() Lua script
  • Consider schedules_natively() returning true for Redis once scheduling is handled inline

Ref: #70, #71

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions