Replies: 1 comment 4 replies
-
|
Looks like upgrading to This might not be a perfect test case of #176 because each target only saves a few KB of data, but maybe a next test could be a version that saves entire |
Beta Was this translation helpful? Give feedback.
-
|
Does |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it will be a minor point release when it happens. Nothing I can see that changes memory handling though. |
Beta Was this translation helpful? Give feedback.
-
|
FYI that pipeline of mine completed successfully. I am rerunning a different version which saves and loads |
Beta Was this translation helpful? Give feedback.
-
That pipeline is just like https://github.com/openpharma/brms.mmrm/tree/main/vignettes/sbc, but with: list(
tar_map(
values = scenarios,
names = tidyselect::any_of("name"),
tar_target(prior, setup_prior(scenario)),
tar_target(rep, seq_len(1000L)),
tar_target(
model1,
run_simulation(
scenario = scenario,
prior = prior,
chains = 3L,
warmup = 4000L,
iter = 8000L
),
pattern = map(rep)
),
tar_target(model2, model1, pattern = map(model1)),
tar_target(model3, model2, pattern = map(model2))
)
)Each of 6 scenarios looks like this:
The whole pipeline has 18012 targets, and 18000 of those targets is 200 MB in memory each. Here is what the memory over time looked like:
Notably, the |
Beta Was this translation helpful? Give feedback.
-
To better diagnose #176, I revisited the
targetspipeline at https://github.com/openpharma/brms.mmrm/tree/main/vignettes/sbc, a simulation study to validate a Bayesian model using simulation-based calibration (SBC) checking.I turned on local memory monitoring using the
log_resourcesargument of the controller. Memory usage of the localtargetsprocess andmiraidispatcher process looks fine:And even after running for 2 days, the dispatcher and pipeline are still running. So that's at least good news.
However, I am seeing a lot of timeouts when the workers are first trying to dial in, and the worker logs look strange. The
targetspipeline tries to keep 50 workers going at a time, and if if I list the number of lines of each worker instance log file, I see:The logs with ~25000 lines are successfully running Stan models:
But the ones with few lines are having trouble dialing in:
I do need to implement #178, but I find it hard to believe this would be a memory issue. Maybe it's because I'm running
nanonext_1.1.1.9015andmirai_1.1.1.9012? I will upgrade and try again. Hopefully the resources on the cluster will still be available.Session info:
FYI @shikokuchuo, @multimeric
Beta Was this translation helpful? Give feedback.
All reactions