Mako Batch/Interactive Templates & Build Fixes

[henryleberre]

The last PR before the Benchmarking one.

Closes #292, #240, and #287.

[sbryngelson]

I think this PR was going to fix the issue of not using multiple ranks with ./mfc.sh test -j X on the Phoenix CPU runner. does it?

[henryleberre]

I think this PR was going to fix the issue of not using multiple ranks with ./mfc.sh test -j X on the Phoenix CPU runner. does it?

I had forgotten to include it. Just did. I added:

               --bind-to none                             \

So when using GT Phoenix, you just need to use -c phoenix while testing and/or running (interactive & batch) MFC.

[sbryngelson]

so the new command to test on an mpirun system is ./mfc.sh test -j X -- --binary mpirun --gpu? this seems to be what I need when testing your PR on Phoenix.

it is also re-building even though the GLOBs are the same?

[henryleberre]

so the new command to test on an mpirun system is ./mfc.sh test -j X -- --binary mpirun --gpu? this seems to be what I need when testing your PR on Phoenix.

it is also re-building even though the GLOBs are the same?

For Phoenix, I would recommend just:

./mfc.sh test -j X --gpu -- -c phoenix

But this also works now:

./mfc.sh test -j X --gpu -- -b mpirun -f --bind-to none

I looked into #288 and it's not quite fixed yet. It is really quite strange. I think it has to do with CMake expecting its binary directory to be empty upon creation and thus skipping GLOB checks within that directory.

[sbryngelson]

so the new command to test on an mpirun system is ./mfc.sh test -j X -- --binary mpirun --gpu? this seems to be what I need when testing your PR on Phoenix.
it is also re-building even though the GLOBs are the same?

For Phoenix, I would recommend just:

./mfc.sh test -j X --gpu -- -c phoenix

But this also works now:

./mfc.sh test -j X --gpu -- -b mpirun -f --bind-to none

I looked into #288 and it's not quite fixed yet. It is really quite strange. I think it has to do with CMake expecting its binary directory to be empty upon creation and thus skipping GLOB checks within that directory.

got it

[henryleberre]

Part of the appeal of this PR (other than there being a template per system) is that:

• Templates are now used for both Batch & Interactive execution. This gives us more flexibility and makes it a lot easier to understand and modify the behavior of these template files. You no longer need to parse through toolchain/mfc/run/engines.py. It doesn't exist anymore!
• Instead of my handcrafted and regex-powered template engine, I switched to an all-around better solution, Mako.
• A lot of indirection was removed (through the support of -- in argument parsing in many places).
• Refactored system() in toolchain/mfc/ so that whenever we execute a command, we no longer depend on the shell. We provide a list of arguments rather a command-string. It's a lot more difficult to create a bug by introducing spaces in a command or variable name.

[sbryngelson]

This looks great - glad to merge once tests pass. We can also now run CPU tests much faster.

[henryleberre]

@sbryngelson I just fixed the rebuild issue! Closes #288.

[sbryngelson]

@sbryngelson I just fixed the rebuild issue! Closes #288.

Thank goodness! It was very strange. What was it?

[sbryngelson]

@henryleberre you have access to NCSA Delta and Purdue Anvil as well. Can you add appropriate batch templates for those computers?

[henryleberre]

@sbryngelson I just fixed the rebuild issue! Closes #288.

Thank goodness! It was very strange. What was it?

There were two issues:

• We only generated the case-optimization FPP files while building, not also while configuring. This causes a GLOB mismatch.
• When I added their generation to the configure step, I had forgotten we also delete the binary directory before running CMake (we have good reasons for doing this). So the files were actually never there while CMake was configuring the MFC targets.

[henryleberre]

@henryleberre you have access to NCSA Delta and Purdue Anvil as well. Can you add appropriate batch templates for those computers?

Could these be added later? It might be a good idea for users to get acquainted with this way of doing things, adding Mako files as they need them.

[sbryngelson]

@henryleberre you have access to NCSA Delta and Purdue Anvil as well. Can you add appropriate batch templates for those computers?

Could these be added later? It might be a good idea for users to get acquainted with this way of doing things, adding Mako files as they need them.

Yes that's fine. Merge when appropriate.