included S3 decorators

[Acribbs]

No description provided.

[Acribbs]

Have implemented functions to allow ruffus interact with S3 buckets using decorators, it downloads a local copy of the file and removes it after processes have been completed. This is the first in a few of upgrades im making to cgatcore to implement cloud functionality and tidy code and improve functionality (kubernetes, Dask, better resource allocation, fault teolerance and cleaning up failed jobs).

Given that I have changed the way decorators are handled in the pipeline/init.py, would someone able to test this branch locally on your code as a sanity check? Tests are passing on actions and works locally for me so should be fine. @snsansom @nickilott @IanSudbery

[IanSudbery]

Excited to see improvements! Happy to help where I can, but isn't the ability to use S3 buckets already baked in? In fact, wasn't it you that implemented it?

[Acribbs]

Thanks. yes, the basic interaction is there for S3 buckets. But when I implemented this it has no real way to check that the files were uploaded to S3 so ruffus has issues tracking the completion of the jobs, so it was very buggy. Here i have implemented features that temporarily download the files, process them and then automatically reupload. I needed to create specific S3 decorators within cgatcore to handle this. This fixes a lot of those issues. Its a start and I will be including other cloud functionality to better support this.