AWS Ec2 launcher

[jdavis-EliLilly]

Working on an AWS EC2 launcher, I have created an initial prototype, that can be modified to be more asynchronous or a great base for future development.

[wlandau]

As I mentioned yesterday, this is a fantastic start! Just knowing how to programmatically work with EC2s and SSH in R (second nature for you) helps me understand what it will take to write a cloud plugin for crew.

The pull request at #17 is great for discussion, and we had a really productive meeting yesterday.

I see some next steps:

1. Allow EC2s to launch asynchronously (non-blocking). crew and mirai already take care of all the connections as long as ephemeral ports are open within a local network that the client and workers share. (No need to expose ports outside the network.) The underlying mechanism is the req/rep NNG protocol where workers dial into the client over TCP. I'm not sure if SSH can play a role here because IIRC it requires a synchronous handshake. TLS in nanonext is pretty much implemented, which should theoretically be just as good.
2. Let crew auto-scale the EC2s. A single pre-launched EC2 is a great start, it really helps break down the problem into smaller pieces. But going forward, I hope crew can auto-scale EC2s up and down based on the task load. All that would need to happen is (1) + a single EC2 per worker instance.
3. One or more separate packages for cloud-based plugins. crew.cluster is for traditional schedulers like SLURM, SGE, etc., which operate very differently. I am not sure yet whether all the AWS plugins should go in a single package, or if 1 package per service would be better. Either way, this separation feels right and will probably reduce the number of package dependencies of each group of plugins.

For the local network requirement in (1), it would be great to have a way to spin up an interactive bastion EC2 running some form of RStudio IDE, and start the crew client/controller from within that IDE. Then crew could launch EC2s inside the same VPC from within the bastion, and those EC2 workers could dial back into the client on the bastion. For me at least, figuring out how to set up that interactive bastion instance seems like the next piece of pre-work.

I am really excited to keep working with you on this!

[wlandau]

An update: since we last spoke, there is now crew.aws.batch: https://github.com/wlandau/crew.aws.batch. I have been using it to run targets pipelines. Planning on an AWS Lambda plugin at some point next year.

[wlandau]

crew.aws.batch is working super well, and I plan to write a Lambda plugin this year.