Skip to content

TrialViewSet.dequeue() has a race condition #267

@carmichaelong

Description

@carmichaelong

Multiple backend workers can pull the same trial since dequeue() is non-blocking, and there are two API tasks running Django for load balancing. This may be causing an intermittent issue (now rarer with less DB pressure) where a trial will appear as an error to a user initially, as one backend worker will fail to upload results (its URL is likely changed with the second worker) but the second one will succeed. This could cause a user to believe their calibration or neutral trial failed due to a setup issue, since the first worker will return an error.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions