ASGS Install on Unity

[notstarboard]

Hello & Happy New Year, Brett, Jason, and others!

I am currently trying to install ASGS on Unity so that we can make use of the URI computing nodes that have recently been installed there. Given that Unity isn't a known environment for ASGS and therefore can't be selected via installer, I just tried to install a "desktop-serial" instance on Unity with the thought that I could manually configure any cluster-specific stuff later. I loaded a few Intel compiler modules ahead of time and overrode the default desktop-serial option (gfortran) to install using intel compilers. init-asgs.sh chugged along for almost two hours but it ultimately bombed out while trying to install the NCO Toolkit. I've attached the end of the console output to this post.

I was wondering if you had any advice for getting this installed, whether it's a different approach to try or help troubleshooting the error. If it matters, I'm not sure if Unity even has a Thredds server set up, so I'm happy for the results stay in the scenario directories for now. Provided I can get through the installation and get the simulations to run, that'd be an awesome start. Thanks!

console_output.txt

[jasonfleming]

Hey @notstarboard great to hear from you! @wwlwpd is the expert here, but I will offer the following suggestions:

1. Add an issue in the ASGS issue tracker with a title like "Add support for Unity at MGHPCC".
2. Create a branch of the ASGS repo with the name issue-xxxx where xxxx is the number GitHub assigned to your new issue from step 1. Making your changes on this branch will help you (a) keep track of them; (b) allow you to commit them back to GitHub on their own branch so that we can see them and provide suggestions; and (c) facilitate a pull request for these changes back to the main repo in the future.
3. @wwlwpd has set up a nice system for adding new platforms with all the settings in one place. Have a look at the README in the platforms subdirectory for the details. Also have a look at the subdirectories in platforms; you'll see that we have used this system to add support for several machines, including lonestar6, hatteras, mike, and penguin. We will also use the method described in the README file to extend support for any/all new HPCs in the future.
4. Check out your issue branch on Unity, make changes under platforms as needed, test, and commit/push your changes on this branch.

As far as nco, it looks like the build is failing because it can't find flex. The flex package may or may not be available on Unity. However, it is a standard package so it should be easy for Unity sysadmins to install it (if you want to ask them). In any case, the nco utilities are handy for post processing tasks, but I don't think they are actually used in the ASGS proper. So you could specify to init-asgs.sh that it should skip the building of the nco package by adding -x "--skip-step nco" when you invoke init-asgs.sh on the command line.

Have fun and let us know how it goes! :-)

[jasonfleming]

I've now updated the README in platforms on the issue-1065 branch.

[wwlwpd]

Thanks @notstarboard and @jasonfleming - another bit of advice I can offer is to take a look at this Dockerfile that's set up to work with Debian; I have not tested it in a while. But if you can get docker installed on Unity (I had to look that up, but appears to be an ubuntu based distro), then you can get access to asgsh via commandline and all the rest. The rub will be when it comes to installing ADCIRC since recently we have not been testing against gfortran for it except in specific cases where noted in the patch set name. Check out, https://github.com/StormSurgeLive/asgs/blob/master/cloud/Dockerfiles/debian/Dockerfile.debian

[notstarboard]

I looked to see if Docker was installed on Unity and it looks like it isn't, but they have a program called Singularity that should do the job:

"This is Singularity, a clone of Docker without sudo privileges.
It should work just the same, but you need to substitute "singularity" for "docker" in your commands"

Would you recommend still trying to run these commands given that I was able to instantiate ASGS?

Edit: After re-reading your next comment, I'm assuming we're probably good without it for now! Looks like a handy program in any case - hadn't heard of docker.

[wwlwpd]

I have looked a little into singularity, and without being able to speak absolutely about it, containers in general seem to be a poor environment for ASGS on HPC because ASGS really needs to be a thin layer between the headnode environment and the overlay of things ASGS needs - ultimately, ASGS (via the ASGS Shell Environment (asgsh)) provides this using just environmental variables. I get the temptation to apply Docker to Singularlty, but the Dockerfile is meant give the ASGS environment on a host that also functions in the same role as the headnode on an HPC system (e.g., a user's desktop or laptop). I can explain more over a call or something. But in general, I will say that running ASGS in a container (Docker or Singularity) on an HPC system doesn't make a lot of sense given the current approach provided by asgsh.

[wwlwpd]

All that's not to say containers can't play a role, I just don't see one in the near term. I am happy to see what you or others come up with, my eyes nowadays tend to be firmly on maintaining a portable and stable system, a lot of times those are opposing goals.

[wwlwpd]

@notstarboard - also we use Docker, mainly targeting desktops, because Ubuntu and other "desktop" Linux's do not come with things like a build environment already there. HPC environments necessarily have all that stuff by user requirement. If MGHPCC has these things already, like even gcc and gfortran then it might be suitable to support directly as a PLATFORM. But if you look at all the stuff that is installed as packages in the Debian Dockerfile, a lot of that are things that we have no compelling reason to support in ASGS - we'd end up having to support have of the fundamental things one can expect to be in a development environment.

Hopefully MGHPCC / Unity has these things already installed. If not, you're really going to want to use Docker because; e.g., we are not going to be supporting the installation of our own gcc. If we went down that road, I'd look at integrating http://www.pkgsrc.org/ which is a fun, but totally out of scope direction.

[notstarboard]

Thank you both for you help on this! I created a branch for the issue on my fork of ASGS, made platforms files for Unity as per the platforms README, and was able to complete ASGS instantiation. Turns out there was indeed a flex module available on Unity and when I loaded that before running init-asgs the NCO errors went away. The modules I loaded were mpich-intel, icc/2022.2.3, and flex.

I've been having no luck getting ADCIRC to build, though. I tried building a few of Dave's custom versions of ADCIRC first, but ran into a bunch of "undefined reference" errors on both attempts. I next tried to install v55.01 to rule out any issues with the custom version and I hit the exact same errors. I've attached the end of the console output as well as the platform files I installed with - would appreciate any suggestions you may have!

init.txt
about.txt
build_adcirc_output.txt

[wwlwpd]

The machine needs a couple of entries in work/cmplrflags.mk, which is what most of the patches we currently maintain in ASGS add. The "name" is the shortened form of the fully qualified domain name usually gotten using the hostname command.

[notstarboard]

I added the FLIBS line to point the compiler at the right NetCDF libraries and I was able to get it to build! I'll keep chugging along - thanks again for your help!

[wwlwpd]

Also, I don't plan on accepting this as an official platform for right now. But would rather flesh it out as a user provided PLATFORM that is made available via ./init-asgs.sh's -p flag, example usage:

  ./init-asgs.sh -p path/to/custom/platformdir

Where path/to/custom/platformdir/myspecificplatform directory exists. It follows the same directory structure as ASGS's ./platforms directory.

This was put in so sites can maintain their own set of platforms without adding it to the list of officially supported machines. An additionall benefit is that sites or individuals can maintain their own repository of sets of platforms

[notstarboard]

That makes sense to me! Totally happy with installing via the -p flag for now.

[notstarboard]

At this point the ADCIRC instance is pretty much all set up! When I run the "verify" command the only things that don't succeed are the TDS connections, which makes sense, since I didn't set those up.

Last night I kicked off a test job that appeared to spin up successfully ("the padcirc.hindcast job appears to have run to completion successfully"), but it's running into issues when trying to download GFS data. There are hundreds of repetitive errors like the following at the bottom of the log file:

2023-Jan-5-T07:53:35+0000 ERROR: get_gfs_status.pl: ftp: Cannot change working directory to '/pub/data/nccf/com/gfs/v16.2': Failed to change directory.

123 downloadGFS /home/joshua_port_uri_edu/asgs-fork/asgs/downloadGFS.sh
2380 main /home/joshua_port_uri_edu/asgs-fork/asgs/asgs_main.sh
[2023-Jan-05-T07:53:35+0000] WARNING: asgs_main.sh>downloadGFS.sh: Failed to get status of GFS cycles with get_gfs_status.pl.

From looking at get_gfs_status.pl this appears to be a path on the NOAA server. Is this probably just an issue with the site hosting the GFS data, then, or can you think of something I may have not done / done wrong in the setup to cause this?

[jasonfleming]

Hey @notstarboard this progress is great see! :-) NOAA rolled out the new version of GFS (v16.3) in December, so the path that ASGS is searching is stale. I actually fixed this in #1057 but it appears to not be in the code for the master branch for some strange reason. Maybe I closed the PR but did not actually merge the fix? In any case, I will fix that. In the meantime, you can add the following line to your ASGS config file to make it look in the right place for GFS data:

GFSBACKDIR="/pub/data/nccf/com/gfs/v16.3"

[notstarboard]

Quick update: that fixed the GFS downloading issue! I'm now about 3-4 hours into the first nowcast on the URI mesh and it seems to be running well so far. Fingers crossed it keeps on rolling!

[jasonfleming]

More good news: @wwlwpd merged the pull request to fix this in the master branch of ASGS. :-)

[notstarboard]

Update on this! GFS data appeared to download successfully and padcirc jobs got kicked off as expected, but these jobs just spun forever, even on really coarse meshes like ec95d. I've been going back and forth with some Unity folks on this, and it seems that we need to build ADCIRC using a specific set of modules in order for Infiniband to be used, so if I build within ASGS it's going to run substantially slower than it should.

The current approach is to try to build ADCIRC on my own and then point ASGS to the appropriate ADCIRCDIR. So far I'm still hitting compiler errors for padcirc/padcswan that folks are helping me with, but once that builds successfully it'll hopefully solve the performance issue. ASGS seems to be working as intended, though, which is why I've been quiet on here!

[jasonfleming]

Hey Josh, great to hear that the ASGS workflow part of this is running as expected. :-) Each HPC has its idiosyncracies, we've definitely had experience accommodating that over the years, especially @wwlwpd. Hopefully it will get sorted quickly.

[notstarboard]

Hi All! I still need to test with the URI mesh and with SWAN enabled, but based on a recent ec95d + WAVES=off test we seem to have resolved the performance issues. With that said, I'm running into errors with my post-processing code that I'm hoping you may be able to help me with.

My code relies on the scipy module, which I have already installed via pip3. However, when I attempt to import that module, I get the following error:

 import scipy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/joshua_port_uri_edu/asgs-fork/asgs/opt/lib/python3.7/site-packages/scipy/__init__.py", line 153, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "/home/joshua_port_uri_edu/asgs-fork/asgs/opt/lib/python3.7/site-packages/scipy/_lib/_ccallback.py", line 1, in <module>
    from . import _ccallback_c
  File "_ccallback_c.pyx", line 210, in init scipy._lib._ccallback_c
  File "/home/joshua_port_uri_edu/asgs-fork/asgs/opt/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

I found this StackOverflow post with a similar error, and based on the replies there I confirmed that libffi-dev is indeed installed on Unity. In case that was a recent change, I also tried the suggestion of uninstalling and reinstalling Python; I ran the following via ASGS:

update-asgs --run-steps python --clean && update-asgs --run-steps python

No dice, though. Do you have any other thoughts on how to resolve this? I figured that I'd start here instead of asking the Unity admins, since ASGS should be using its own Python instance instead of the one installed on Unity. Perhaps that libffi.so file needs to be made available to ASGS's Python somehow? Thanks!

[jasonfleming]

Hey Josh it looks like your script is using the Python3 environment installed by ASGS, which is good. But yes you are right, it also means that the Python modules installed on Unity won't be helpful. My guess is that you could modify $SCRIPTDIR/cloud/general/init-python.sh to add modules needed for post processing. I don't have a lot of experience with the python subsystem in ASGS, so please give that a shot and repeat your previous command with --run-steps python and let us know how it goes.

[notstarboard]

Thanks for the suggestion! Unfortunately I don't think it's a question of importing modules - from searching around, I think the issue is that the Python environment isn't able to find libffi.so when it builds, which results in a silent failure that causes it not to find the _ctypes module. If I try to run "pip3 install ctypes" it gives me an error...

ERROR: Could not find a version that satisfies the requirement ctypes (from versions: none)
ERROR: No matching distribution found for ctypes

...and if I try to "import ctypes" or "import _ctypes" it gives me the same "No module named '_ctypes'" error.

I'm not convinced the update-asgs command above is actually wiping and rebuilding the Python environment like I was expecting, which complicates things. One forum post suggested to switching to Python 3.7.13 because they included some fixes relating to ctypes, but when I update the PYTHONVERSION in init-python3.sh to 3.7.13, run the update-asgs commands above to update the python environment, and then run "python3 --version" from within ASGS to check the version, it still says 3.7.9. There could be other places the version is specified, though; I still need to poke around.

I've been trying to get help from some folks on the Unity side too, and their suggestions were along the lines of "reinstall ctypes in your Python environment". From what I can tell, though, ctypes is included in any base install of Python3; it doesn't need to get installed separately, and moreover the standalone version was written for Python 2.x and hasn't been updated in 15 years. I think the right libraries just need to be available in order for the _ctypes stuff to work correctly. The right libraries exist on the base Unity system, I don't really know how to also make them available to ASGS's virtual Python environment. asgs/opt/bin is missing libffi.so out of the box, which I think is the key file, but copying over the one from Unity and running the commands above don't solve the issue. I'm not sure if there's more to be done to "install" libffi-dev for the virtual Python environment, or if it's some other issue that doesn't surface as frequently on StackOverflow.

At any rate, I'll keep fighting with it and let you know if I make any progress!

[notstarboard]

Ah, well I immediately upon posting realized the command above needed to be amended to "python3" instead of python. This will be helpful! I'll give rebuilding it for real another shot.

[notstarboard]

I'm not sure which of the things I tried did it, but after rebuilding for real we're in business! I can run "import ctypes" with no trouble now. I'm first going to test the post-processing code outside of ASGS and then, once I'm sure it's working, I'll get back to performance testing. Onward!

[jasonfleming]

That sounds great, this is the "confessional" method of debugging. :-) Let us know how it goes!

[notstarboard]

The post-processing code does indeed work both inside and outside ASGS! So, I'm back to performance testing. And, as is tradition, this brings me to another question:

Kevin (Unity admin) suggested I run ADCIRC with the following flags: "--exclusive", "--mem=125g" (125g being the actual cap for a job on the URI nodes), and "--nodes=2" (or 3, or 4... the idea is to test and see how it goes). His thought is that it will be better for performance partition-wide, and also for the jobs themselves, if they're the only things running on each node they use.

Is there a way to implement this in ASGS? I know we typically configure things at the CPU level with NCPU and NCPUCAPACITY, but I've never attempted this at the node level. Would it be enough to update the JOBLAUNCHER environment variable (currently just 'srun ') with these three flags and to comment out NCPU and NCPUCAPACITY in the config file?

Also, from reading through the wiki and some config files to see what I could dig up on this, I encountered the NUMWRITERS flag. So far I haven't been using this, but I do see that some other config files might have 1, or a few, processors dedicated to this. Do you have any general advice as to when this is a good idea? Figure I may as well squeeze out every bit of performance while I'm down this rabbit hole :)

[wwlwpd]

NUMWRITERS is generally, always "1" - I never benchmarked this, but I seem to recall performance degrades as this goes up to do IO/bottle necks. I could be wrong.

For performance testing, it might be a little awkward to do lots of testing using ASGS because we just use one queue script template; but you can create your own and use it by specifying the QUEUETEMPLATE (something like that, sorry not looking at the rarely used) variable. If you're testing scaling based on limiting the number of processors per node, you may want to fiddle with PPN.

That said, having done a ton of benchmarking over the years with ADCIRC; I would want a less awkward set of tools/means in ASGS to do this kind of thing. Let me know what your scaling experiments look like, and maybe I can advise what tools we need or how I'd approach it. Essentially, you're looking at:

foreach NCPU,
  foreach PPN
    * adcprep
    * run padcirc
    * validate run
    * collect timing info
  endfor
endfor

But it would be neat to have a way to describe a "plan" that consisted of NCPU, PPN, and a custom queue template that had the memory resource bits able to be fiddled with; that when run would collect the numbers and do all that stuff mentioned above. Let me know what you're needing, maybe I can help add this to ASGS.

[notstarboard]

That's very good to know! I'll plan to use 1 going forward as well. Am I correct that I would need to create my own qscript file and update QSCRIPTTEMPLATE if I wanted to run in the exact way the admin recommended? NCPU and PPN do seem like a possible workaround though. I know that the nodes on Unity have different amounts of CPUs on them, but I could specifically set PPN to be the amount of CPUs on the beefiest nodes and then have NCPU be some integer multiple of PPN. Hopefully then SLURM would be smart enough with the base qscript file to feed the jobs to the nodes with enough CPUs and therefore use the whole node.

I do have a way of testing with a standalone ADCIRC run, so at least while I'm just performance testing I might roll with that!

[notstarboard]

I'm actually struggling with the standalone run - things have changed since that run, and I'm running into module-related issues again - so I'm just rolling with ASGS. I think I have found a way to use these flags based on your suggestions, though. I basically just set PPN and NCPU such that I'd be using 64 CPUs per node (which is the whole node for the biggest URI nodes), and then customized the qscript file to specifically use the --exclusive flag (just in case) and --mem, as you alluded to. I've confirmed that this works again for ec95d + SWAN - a forecast took like 3 minutes, feels reasonable - so now I'm trying it again with URI mesh + SWAN.

I used NCPU=255, NUMWRITERS=1, NCPUCAPACITY=256, and PPN=64, and when I run squeue I can confirm that exactly 4 nodes are being used. So, hopefully things are working as Kevin suggested now! If things are still slow I'll continue to troubleshoot with the admins. Thanks!

[wwlwpd]

Thanks for the update! I tend to cheat, you can reuse and old rundir and just modify as needed but jumping into it and manually submitting the queue scripts. It's how I debug things when they go awry. At the end of the data ASGS is just doing what you'd do by hand, but repeatedly. I know that probably doesn't help. It might also help that there is way to tell ASGS to run the hindcast portion once, then end:

HINDCASTONCE_AND_EXIT=yes

I have not tried this out for benchmarking, but conceivably this could be used to script the values in the configuration file (PPN, etc) then just loop loop around some number of PPN and NCPUs, updating the config file, then doing run.

But I had not considered using that for benchmarking, it might provide a nice way forward for such activities.

[wwlwpd]

recorded idea to script benchmarking by configuration file manipulation, #1080

[jasonfleming]

@notstarboard great to hear that the post processing is working! :-) Where did they suggest that you supply these flags? In the queue script itself? Or on the command line sbatch command?

[notstarboard]

Sorry for the slow reply here! The admin didn't specify but I'd expect these would be part of the sbatch command, since the main goal is just to run ADCIRC using these flags.

[notstarboard]

This weekend I was finally able to complete a benchmark run with SWAN and one of the RICHAMP project meshes! I had been fighting with seg faults for all larger meshes too, but "ulimit -s unlimited" ultimately got me past those. The padcswan runtime for the 5-day forecast on 256 CPUs was 2:02:48, which was better than I was expecting. Deb had done a similar benchmark with 360 CPUs on Hatteras and it took 7-8 minutes longer despite the 104 extra CPUs! The combination of the new nodes on Unity, running exclusively ADCIRC on any nodes used, and adding that one writer CPU made a pretty big difference.

I think we can finally consider this discussion complete. Thanks as always for all of your help :)

[notstarboard]

@jasonfleming I saw your post on the other issue - thanks for checking in! I had been sidetracked for a while with more post-processing related things (all on the richamp-support repo), but I'm getting back to this. There are two things I need to accomplish before we're basically ready on our end for a full live test, including dashboarding (probably in a few months - RIEMA is trying to schedule this but they're pretty jammed through the summer).

1. I'm currently having issues with the NetCDF output when I run with the URI mesh. ASGS gives an exit code of 0, I haven't found anything worse than warnings in the ADCIRC log files, and the simulations do seem to run to completion, but when I open the output files, anything with a time dimension is an empty set. So, for example, the fort.63 would have the x, y, and element fields populated as expected, but time is just [] and so is inundation.
2. Since Unity doesn't have a THREDDS server, the solution the admin suggested is to upload the files to an Amazon S3 bucket he created for this purpose. I've looked at this enough to determine that it should be doable via bash, but I haven't paid the time tax for this yet.

I still have some things I can try before I bug you both for help, and Dave Ullman is probably the first guy to talk to anyway since he got this working on Hatteras! So, unless you have a really good idea on either of these pieces off the top of your head, I'll just keep plugging away and keep you posted.

[jasonfleming]

Hey @notstarboard great to hear from you! Yes, here are some ideas:

1. The ADCIRC process exit codes are unfortunately not a reliable indicator of success or failure of a run. The ASGS is able to report some types of failures (e.g., numerical instability), but there are many things that can go wrong with ADCIRC that the ASGS cannot detect. Some investigative work in the ADCIRC log files (including subdomain log files) is probably the best next step.
2. I use s3cmd to upload and download files to/from S3 storage. It is easy to set up (just unpack the s3cmd files into ~/bin) and works very well. I think you might also be able to use the minio client, although I have not tried that. Whatever approach you try, please let us know how it turns out!

Enjoy. :-)

[wwlwpd]

I do not recommend using s3 to do anything productive where interactive human effort is concerned, it's incredibly inefficient for this process because you have to futz with non-traditional utilities and JSON API clients; it's built for machines not humans. I would be surprised if y'all don't have access to a simple anonymous FTP (read only, anyway) that lets users copy files internally. This part of the process has been wrought with over complication over the last decade, and it's really unfortunate. Anyway, find a place that can expose either an public anonymous FTP or directory listing over http/https. An alternative is some place like mega.nz or worst case, dropbox.

[wwlwpd]

My comment above applies for interactive/iterative collaboration between people. Not supporting uploading of files. If you want to automate that, then this would be done as part of a POSTPROCESS script which I think maybe you're looking to do. It remains the case that pointing an end consumer to an s3 bucket rather than an ftp site or http with directory listing is ill advised. I don't even know what their brower listing might look like, but I assume it's not any better. If you can get a VM or something up to run THREDDS, it's straightforward to get it going; a catalog of files is what THREDDS is meant to do - there are no good or modern replacements for what it does, and I've become increasingly assured that THREDDS (and some OPeNDAP provider for subsetting, which I don't have any experience with) is a valuable piece to our infrastructure. The modern stuff is fine for automation, but when people are involved they straight up suck.

[wwlwpd]

Last reply I promise - I forgot to add setting up THREDDS on a VM is as easy as running a docker container. But then your center might do something silly, like most centers, which disallows docker in favor of singularity. Again, more unproductive work and zero benefit. I'm still happy to help, but the point of my soapbox here is that it doesn't have to be as hard as they make it. So push hard on what is possible and they might still be willing to help you out in a humane way.

Update: if you can get a VM that exposes THREDDS on 80/433, then it's also very easy to get minio going as @jasonfleming mentioned. You might need the admins to open up ports 9000-9010, but this gives you an S3 (API) interface on the same data that humans can navigate via THREDDS. This is a win-win. Minio is really in our case just for reading. Upload should still be done via scp or rsync. Backblaze is another service that offers s3, but also has listings. It's a to cheaper than AWS and storage is all they do. I would never use AWS for any kind of storage for a variety of reasons, but the most practical is that they simply do TOO MUCH STUFF. 😄

[jasonfleming]

Hey all, @wwlwpd has a good point: S3 vs THREDDS really depends on what you plan to do with the data once you've uploaded it. My response above was just related to automating the upload to S3 with a bash script without a thought about what happens to the data once it gets where it's going.

@notstarboard can you give some detail about the fate of the uploaded data? Will there just be a subsequent automated download to the RICHAMP GIS stuff? Is the S3 or THREDDS server supposed to be archival storage? Is there an audience for the raw netCDF data that you will want to give interactive access to?

[notstarboard]

I appreciate all of the thoughts here! I don't know Kevin's reasons for suggesting S3, but I can share your suggestions after I give you a chance to reply to this post and then see what his take is. I do know that he wasn't very familiar with THREDDS, so some of it could be just trying to keep things simple by sticking to things that other groups are already using.

The basic need we have is an intermediate place to store files that collaborators on the RICHAMP project will use to drive dashboards and such for RIEMA. Chris Damon (collaborator working on the dashboarding) has consistently said he needs some sort of public URL for this. I don't know what tools he's using to check for new files, but I've brought up him just grabbing them directly from Unity and it seemed to be a nonstarter.

This intermediate location doesn't necessarily need to be a long-term storage location, as we'll still have the local runs done on Unity. We'll probably want to have some sort of backup location with massive amounts of storage to archive past forecasts, as I believe Isaac does want to archive them and I can see the utility in that. I suppose long term this could be S3, or some other volume on Unity, or something else entirely. It's a less pressing concern than just finding a location that all URI teams can access that allows for simple uploading and downloading of files.

End users will only be interacting with this data via dashboards as far as I know, so they will have no need to get at the underlying NetCDFs.

---

I did give s3cmd a shot and it was easy enough to get going! I was able to upload a test file, anyway. It should be easy enough to build into ASGS as a POSTPROCESS task as Brett mentioned.

At a minimum I need to check in with Chris Damon again about how he wants to access the files, for example, is it best to just have one path and I'll overwrite any existing files there whenever we run a forecast, or should I create some sort of unique folder structure like ASGS does with THREDDS? If the former, could I just scp it to the actual final destination for these files instead to save a few minutes and cut out the middle man?

[jasonfleming]

@notstarboard you have an option we haven't mentioned. You could ask the Unity admins to set up a simple web server with nginx or similar with directory listing turned on and then scp the files to that server. That's probably the simplest solution possible.

The last paragraph above gets to another issue: how do you notify Chris that new results are available, and what those results contain (e.g., what advisory was used, etc). We use email for that. Chris can also poll the server and parse metadata to see if there is new data.

[notstarboard]

Hi All! Update on the above - I was able to convince Chris to get an account on Unity so that I can just move the files to a staging directory via cp and he can then use scp to move them wherever he would like. While testing with S3 it quickly became clear that upload times were going to be a concern, so this was definitely the new path of least resistance.

So, everything is pretty much working at this point - all that's left is more testing. I'm currently working on some hindcasts for Henri that RIEMA will use during a tabletop exercise next month to test out the system. I do have one question related to that, but I'll post that to its own discussion as it's not specific to Unity.

[jasonfleming]

Thank you for the update @notstarboard that is great to hear! Please let us know how the tabletop exercise goes, these things are always valuable for readiness and tend to generate a lot of lessons learned that would be of interest across the board. :-)