How should one monitor Queue Depth?

[zachgersh]

Hey river friends,

I know we've got Subscriptions for monitoring what's happening with individual job(s) but I really would love to know how to monitor queue depth? Metric is helpful for knowing if workers are processing your queue fast enough (among other things).

[brandur]

If you have SQL access, it should basically look like running SELECT count(*) FROM river_jobs WHERE state = 'available'.

May be something we want to add to the Go list API as well (i.e. allow for a count as well as finalizing a response list), but I don't think it's in there right now.

[zachgersh]

@brandur I think what I want to do is leverage this

river/riverdriver/riverpgxv5/river_pgx_v5_driver.go

Line 116 in 0f09f80

func (e *Executor) JobCountByState(ctx context.Context, state rivertype.JobState) (int, error) {

(its a close enough approximation). Using the river provided SQL API at least insulates me (hopefully) from table changes y'all may want to make directly (like renaming the whole table for instance).

Would love a way to subscribe to overall river cluster stats but I get y'all may not have 100% worked that out yet.

What do you think of this is an initial idea?

[brandur]

See the note at the top of that file, but the driver API should be considered highly unstable and may be broken at any time (it has been many times in the past). If you're okay with that, it's fine, but also the name of river_job is quite unlikely to change at this point, so I'd guess that using your own SQL would be the more likely option not to break.

Would love a way to subscribe to overall river cluster stats but I get y'all may not have 100% worked that out yet.

Yeah, currently subscriptions are for only what's going on in the River client running on that specific node. Maybe that changes at some point, but I didn't have any immediate plans to bake it in. Some kind of "overall stats" API for a River client isn't a bad idea though (for logging purposes, etc.). I could see that coming in.

[zachgersh]

Cool, makes sense. Yeah, even if the package is highly unstable, upgrading between river versions (if you change the function signature) should break my app. I can also write some integration tests around this as well to make sure I get consistent counts back. Easiest way without an official stats API to make sure I break if y'all change the logic.

[bgentry]

I’d also agree that the database schema is much more stable than the internal driver interface. Simple stuff like counting jobs by state or queue are pretty much guaranteed not to change ever. There’d be some pretty major migrations to go along with it if they did.