Replies: 1 comment 5 replies
-
|
another way to see it is through average fit times as a fn of workers tibble::tibble(
time = c(cumsum(rep(200, 10)), cumsum(rep(515, 5)), cumsum(c(rep(1000, 3), 200))),
fit = rep(c(1,2,3,1), times = c(10, 5, 3,1)),
workers = rep(c(1,2,3), times = c(10, 5, 4))
) |>
dplyr::mutate(
i = cumsum(fit),
.by = workers
) |>
ggplot(aes(x = i, y = time, colour = factor(workers))) +
geom_point() +
geom_line() +
scale_x_continuous(breaks = seq(10)) +
labs(
x = 'Total Fits',
y = 'Cumulative Time',
colour = 'Parallel Workers'
) +
theme(
legend.position = 'top'
)
|
Beta Was this translation helpful? Give feedback.
-
|
How many cores does your local machine have? If Stan is already running with |
Beta Was this translation helpful? Give feedback.
-
|
machine has 16 cores and it looks like the 12 are being engaged properly based on But as you see from the plot output even with 2 parallel runs efficiency is below compared to a sequential performance. |
Beta Was this translation helpful? Give feedback.
-
|
Since you're running this locally, how does performance compare to using library(mirai)
daemons(n = 3)
task1 <- mirai({...})
task2 <- mirai({...})
task3 <- mirai({...}) |
Beta Was this translation helpful? Give feedback.
-
|
after asking more q's to devops it seems like there was an unwritten constraint on users that caps the number of cores a user can use on the shared k8s instances, which explains why I was getting these results. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, glad you figured it out. |
Beta Was this translation helpful? Give feedback.
-
I am not sure which bucket this falls into...
I have a joint model in stan (brms) that takes ~400s to warmup and sample in an interactive session.
I am running a kfold (k=10) with {rsample} to evaluate posterior predictions.
soobject to avoid recompiling on each foldIf I try to run it with crew for 1 of the folds in a child session it retains the performance (~400s)
But when I try to use a controller in targets that fits 3 folds at a time
performance drops to ~30min warmup+fit, where most of the drop is in the warmup.
I have tried to do more brms pre-processing and am extracting the diagonal mass matrix from a single chain warmup only run to pass in the inv_metric on the sampling step.
When I look at the stderr/stdout it looks like everything is running ok in the child sessions
what that looks like
In the interactive session run that drops my sampling times to ~200s, but I am seeing a similar loss of efficiency in the targets run ~15min.
I am still trying to find a way to reprex this in a canonical way but wanted to ask in the meantime if you have seen something like this previously and know if this is just resources on the instance maxing out and stan can't be run in parallel locally like I am pushing my machine at the moment.
Beta Was this translation helpful? Give feedback.
All reactions