Dynamic Smagorinsky on HPC

[SamDeAbreu]

Hi all,

I see that there has been development by @tomchor and others on a Smagorinsky closure but with a dynamic coefficient, which is amazing! I've been able to run the directionally-averaged version locally, but run into issues when running it on my school's HPC (version 0.95.10). Specifically, the code stalls on building the nonhydrostatic model.

It appears this is a known issue (#3642 (comment)) when it was being tested on a server. I was wondering if the cause has been determined yet? Or if we're still not sure?

Thanks,
Sam

[tomchor]

I think we never understood why that compilation wasn't working on sverdrup. When we switched to using Github's machines (#3926) it worked pretty well, so I think compiling the dynamic coefficient is just too much for some machines... ?

That said, it takes a looong time to compile. How long have you waited for before throwing in the towel?

[SamDeAbreu]

I've waited up to 30min trying to build a 64x64x64 model, but the HPC kills it/runs out of memory before anything else happens. Very weird that I can build the 64x64x64 model on my okay local GPU in a time <5min, but not on a more advanced GPU on an HPC.

[tomchor]

Does it build on a CPU on the same HPC machine?

Also, can you share details on the machine?

[glwagner]

You may not be requesting enough memory on the node, which is required for compilation. Also, the CPU may be slow even if the GPU is more advanced. Finally you may also want to try using all of the threads on the node you're on (eg many CPUs), because I believe that this will allow you to benefit from multithreaded compilation.

Note that setup is basically compilation and so it doesn't have to do with the GPU you're using, only the CPU(s) that you're doing the compilation on.

[SamDeAbreu]

That appeared to be the issue. I wasn't aware that building the model was CPU intensive. After requesting more CPUs for the job in addition to the GPU, it built the model and ran. Thank you guys!