Launchers in mirai 0.9.1

[shikokuchuo]

mirai 0.9.1 (newly on CRAN!) implements launch_local() and launch_remote().

They could be used directly by crew if it passes the required environment variables on the command line instead of via the wrapper crew_worker(), i.e. in the manner of:

CREW_LAUNCHER=a CREW_WORKER=b CREW_INSTANCE=c Rscript -e 'mirai::daemon("ws://[::1]:11111")'

The shell command can be generated by:

launch_remote("ws://[::1]:11111", rscript = "CREW_LAUNCHER=a CREW_WORKER=b CREW_INSTANCE=c Rscript")
[1] "CREW_LAUNCHER=a CREW_WORKER=b CREW_INSTANCE=c Rscript -e 'mirai::daemon(\"ws://[::1]:11111\")'"

Just a thought.

Given the oddities I found with processx when debugging TLS on MacOS, I am in favour of stripping away the additional dependencies if it gives us additional robustness.

An alternative you may consider is using: ps::ps_mark_tree() and related functions to keep track of child processes. I wonder if the functionality there will be sufficient.

[wlandau]

Awesome! So glad to see mirai 0.9.1 on CRAN! And thank you for all the promising leads. I hope to make time for all this tomorrow, as well as strictly depending on mirai 0.9.1 to tighten up TLS support in crew.

[shikokuchuo]

The oddities I am referring to are those I found for the closed issue 65 for mirai. That in particular is not a concern - as I mention - it is pointless to use TLS for local processes. However there is no telling how else it might interfere.

[shikokuchuo]

The passing of environment variables on a shell seems robust to me - don't see how it would be platform dependent?
Also I thought speed wouldn't be an issue as it is used as a last resort to kill workers.

I am probably biased towards reducing dependencies though.

[shikokuchuo]

Also I came across this: https://github.com/coolbutuseless/deft

I haven't used it yet myself, and it may be a bit much for {crew} to carry compiled code just for this. But it would remove the dependency on data.table, which seems a bit heavy just for rbindlist().

Just throwing some ideas around.

[wlandau]

Hmm... somehow I completely missed shikokuchuo/mirai#65. Thanks for mentioning it.

I wonder if all of processx/callr would be a problem, or if it there is a feature we could disable, i.e. connections/polls that processx creates.

The passing of environment variables on a shell seems robust to me - don't see how it would be platform dependent?

If the mechanism were the actual environment variables themselves, then that would be fine. But when I tested it by manually, starting different R processes and setting environment variables there, the functions in ps could not find the processes. The mechanism seems to be the ENV=VALUE prefix in front of the text command of a process, and I can't say how this text would look or if it would even appear on non-Unix systems.

[shikokuchuo]

If the mechanism were the actual environment variables themselves, then that would be fine. But when I tested it by manually, starting different R processes and setting environment variables there, the functions in ps could not find the processes. The mechanism seems to be the ENV=VALUE prefix in front of the text command of a process, and I can't say how this text would look or if it would even appear on non-Unix systems.

Just occurred to me that this might be an inheritance issue - perhaps try export ENV=VALUE. I think ps directly calls fork and execve but I am not really sure what R / system2 does.

[shikokuchuo]

I realise the problem with using env vars is more fundamental to R, as I was implementing passing the libpath to dispatcher. Luckily for that purpose there is a well-documented base function, but just relying on env vars would not have been robust. And probably wouldn’t work on Windows (yes, remember that?!)