Disconnected operation of a controller

[billdenney]

I have an idea that would take a different form from #106, but could possibly accomplish a similar purpose.

Use case 1: I want to be able to close my laptop and have jobs continue to run on a computational cluster
Use case 2: I want to use my laptop to send jobs to a cluster where I may then have intermittent access (e.g. I'm flying on a plane)

If there were some form of store-and-forward operation at the controller level, then disconnected operation like the above could become feasible. It would also enable multi-hop topologies as in the diagram in #106.

The concept that I'm thinking of is that crew controllers could have scratch space to store results locally. Then, the user could connect intermittently to the controller and ask "do you have any results in your scratch space to send me?" If the user is already connected (i.e. how crew operates now), then results would be sent back immediately.

I see important complexity overhead here that should not be underestimated, but at the same time, the benefits could be significant in terms of allowing different topologies (as in #106).

[wlandau]

If the connection is unreliable, it would be much better to log directly into the login node of the cluster and run the pipeline from there.

In fact, taking a step back here, I hope to make this sort of situation easier to navigate using cloud storage in targets. As of ropensci/targets#1130, all the important data seamlessly backs up to a bucket, even the metadata. So you can more easily run the pipeline on one machine and view the results on a different one. I think this pattern is likely to be more comfortable and have stronger network security than #106.

[billdenney]

That makes sense. When running from the login node, is it valid to copy the contents of the _targets directory back to a local computer (meaning a laptop or similar) for subsequent work? (That could be operationally the same for my use cases.)

[wlandau]

Yes, if _targets/ is not too big and external file targets are not a barrier for what you want to do locally.