Troubles with throughput/backlog tests

[wlandau]

Following up on shikokuchuo/nanonext#24 (comment): I just retested https://github.com/wlandau/crew/blob/46e9fd2b37d888d5f27e01c6c0aec48c0f4f551c/tests/throughput/test-backlog-tasks_max.R on my local Ubuntu machine. Using crew version 0.6.0 (current CRAN), nanonext 0.10.4.9024, and mirai 0.11.2.9017, 28 out of 60000 tasks hung unresolved. Currently testing if I get the same result without TLS.

[wlandau]

Yup, it's also happening without TLS. The test code for that is:

library(crew)
controller <- crew_controller_local(
  workers = 20L,
  tasks_max = 100
)
controller$start()
names <- character(0L)
index <- 0L
n_tasks <- 60000L
system.time(
  while (index < n_tasks || !(controller$empty())) {
    if (index < n_tasks) {
      index <- index + 1L
      cat("submit", index, "\n")
      controller$push(
        name = as.character(index),
        command = TRUE
      )
    }
    out <- controller$pop()
    if (!is.null(out)) {
      cat("collect", out$name, "\n")
      names[[length(names) + 1L]] <- out$name
    }
  }
)

It hangs after a bunch of "collect ..." messages at the end. After interrupting the loop, purrr::map(controller$tasks, ~.x$data) reveals 2 unresolved mirai() tasks out of 60000.

[shikokuchuo]

As per my comment here shikokuchuo/nanonext#24 (comment) your tests run fine on my Ubuntu 22.04 setup. The non-TLS example you posted above also completes fine. It might be an idea to try installing a clean version of R - perhaps pull the R-devel svn and build it, if that machine is behaving idiosyncratically.

[wlandau]

Yeah, I will need to look into that. Everything else about the crew/mirai/nanonext integration in dev looks great. Feel free to release those next versions of nanonext and mirai you mentioned. crew is almost ready to go (just some interface encapsulation stuff to work out in #137, and then I will consider it feature-complete).

[shikokuchuo]

Thanks for letting me know! That's great that the integration is going well. I'm afraid I haven't been able to track down a possible cause of this issue yet. Tests fine on a couple of other machines (workstation, and a very old 2-core device) so I don't believe it is related to hardware constraints.

[shikokuchuo]

I believe this is fixed in shikokuchuo/mirai@d7b4f59. The package had certain safeguards that prevented this type of error (but perhaps not always), but I have re-implemented this part so now everything works without those safeguards, and I presume this error should also go away.

It is somewhat non-trivial combining a language with precise memory control such as C and the garbage collection in R, a trade-off between efficiency and safety. But overall I think performance and memory utilisation improves as a result.

[wlandau]

I believe this is fixed in shikokuchuo/mirai@d7b4f59. The package had certain safeguards that prevented this type of error (but perhaps not always), but I have re-implemented this part so now everything works without those safeguards, and I presume this error should also go away.

Thanks, you did indeed fix the issue I. All my crew tests run reliably now.

It is somewhat non-trivial combining a language with precise memory control such as C and the garbage collection in R, a trade-off between efficiency and safety. But overall I think performance and memory utilisation improves as a result.

That makes sense, given how nanonext exposes C-level NNG structures directly to R.

[shikokuchuo]

That's great! This certainly wasn't an easy one to figure out. But now that I have, it will make our packages that much more robust going forward.