Discussion of future enhancements

[dtenenba]

There are several areas where we could greatly improve Motuz. This work can be divided into chunks, but IMO should all lead toward these goals:

• Web app runs in Open OnDemand. Advantages are:
  • OOD takes care of authentication; auth can be removed from motuz web app
  • We are on a node that can submit HPC jobs w/o SSH
  • We are on a node that has all needed filesystems mounted already
  • Motuz can either run as a Passenger app (runs on same node as OOD) or as
    an interactive app/plugin (runs as an HPC job on a compute node, meaning the user has to start it up whenever they want to use it. In either case the app will run as the user
    so it will have the appropriate permissions out of the box, no need for sudo.
• Copy/hashsum jobs run as HPC (Slurm) jobs. Advantages are:
  • Jobs run on separate nodes so web app can be updated without interrupting copy jobs.
  • Slurm can serve as the job queue/manager, so celery can be removed.
  • Improved performance - not everything is running on a single machine anymore.
  • The only dependency needed on compute nodes will be rclone.

There is some discussion of the latter goal here.

There is already a branch that has authentication removed. It runs as the user you run it as....No login needed. Intended only for use with OOD.

There are some interim phases that may be worth considering:

• Modify motuz so that all commands (ls, etc) are run on a remote machine via SSH. Advantages:
  • Web app can run anywhere (including Docker Swarm or k8s), does not need
    specific file systems mounted.
  • No longer need specially configured/fast/bare-metal node to run motuz. Cattle not pets.

The two bullet points at the top can be accomplished separately. And probably should, as a way of dividing up the work. But it makes sense to do them both.

Other features that have been suggested (may create separate issues for these, but this issue is for wide-ranging discussion):

• Ability to display contents of files (using rclone cat under the hood).
• Ability to download files to local machine
• Ability to upload files from local machine (?)
• Ability to delete files/directories
• Ability to use credentials in ~/.aws directory
• Along w/the above, support for IAM role chaining

I will report progress on these changes below.