Skip to content

Commit

Permalink
Website small tweaks and improvements and doc syntax fixes (#362)
Browse files Browse the repository at this point in the history
  • Loading branch information
sbryngelson authored Mar 3, 2024
1 parent cfcfb66 commit 605ee69
Show file tree
Hide file tree
Showing 8 changed files with 122 additions and 124 deletions.
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,9 @@ if (MFC_DOCUMENTATION)
install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/docs/res"
DESTINATION "docs/mfc")

install(FILES "${CMAKE_CURRENT_SOURCE_DIR}/docs/robots.txt"
DESTINATION "docs/mfc")

install(FILES "${CMAKE_CURRENT_SOURCE_DIR}/docs/index.html"
DESTINATION "docs/mfc")
endif()
Expand Down
16 changes: 7 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
</p>

Welcome to the home of MFC!
MFC simulates compressible multi-component and multi-phase flows, amongst other things.
It scales <b>ideally to exascale</b>; [tens of thousands of GPUs on NVIDIA- and AMD-GPU Machines](#is-this-really-exascale), like Oak Ridge Summit and Frontier.
MFC simulates compressible multi-component and multi-phase flows, [amongst other things](#what-else-can-this-thing-do).
It scales <b>ideally to exascale</b>; [tens of thousands of GPUs on NVIDIA- and AMD-GPU machines](#is-this-really-exascale), like Oak Ridge Summit and Frontier.
MFC is written in Fortran and makes use of metaprogramming to keep the code short (about 20K lines).

Get in touch with the maintainers, like <a href="mailto:shb@gatech.edu">Spencer</a>, if you have questions!
Expand Down Expand Up @@ -96,7 +96,7 @@ They are organized below, just click the drop-downs!
* Multi- and single-phase
* Phase change via p, pT, and pTg schemes
* Grids
* 1-3D Cartesian, Cylindrical, Axi-symmetric.
* 1-3D Cartesian, cylindrical, axi-symmetric.
* Arbitrary grid stretching for multiple domain regions available.
* Complex/arbitrary geometries via immersed boundary methods
* STL geometry files supported
Expand Down Expand Up @@ -162,12 +162,10 @@ If you use MFC, consider citing it:

## License

Copyright 2021-2024.
Copyright 2021-2024 Spencer Bryngelson and Tim Colonius.
MFC is under the MIT license (see [LICENSE](LICENSE) file for full text).

## Acknowledgements

<p align="justify">
MFC development was supported by multiple current and past grants from the US Department of Defense, National Institute of Health (NIH), Department of Energy (DOE), and the National Science Foundation (NSF).
MFC computations use OLCF Frontier, Summit, and Wombat under allocation CFD154 (PI Bryngelson) and ACCESS-CI under allocations TG-CTS120005 (PI Colonius) and TG-PHY210084 (PI Bryngelson).
</p>

Multiple federal sponsors have supported MFC development, including the US Department of Defense (DOD), National Institutes of Health (NIH), Department of Energy (DOE), and National Science Foundation (NSF).
MFC computations use OLCF Frontier, Summit, and Wombat under allocation CFD154 (PI Bryngelson) and ACCESS-CI under allocations TG-CTS120005 (PI Colonius) and TG-PHY210084 (PI Bryngelson).
50 changes: 25 additions & 25 deletions docs/documentation/case.md

Large diffs are not rendered by default.

68 changes: 32 additions & 36 deletions docs/documentation/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
You can either download MFC's [latest release from GitHub](https://github.com/MFlowCode/MFC/releases/latest) or clone the repository:

```console
$ git clone https://github.com/MFlowCode/MFC.git
$ cd MFC
$ git checkout <release tag>
git clone https://github.com/MFlowCode/MFC.git
cd MFC
```

## Build Environment
Expand All @@ -21,24 +20,24 @@ Please select your desired configuration from the list bellow:
- **On supported clusters:** Load environment modules

```console
$ . ./mfc.sh load
. ./mfc.sh load
```

- **Via [Aptitude](https://wiki.debian.org/Aptitude):**

```console
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install tar wget make cmake gcc g++ \
sudo apt update
sudo apt upgrade
sudo apt install tar wget make cmake gcc g++ \
python3 python3-dev \
"openmpi-*" libopenmpi-dev
```

- **Via [Pacman](https://wiki.archlinux.org/title/pacman):**

```console
$ sudo pacman -Syu
$ sudo pacman -S base-devel coreutils \
sudo pacman -Syu
sudo pacman -S base-devel coreutils \
git ninja gcc-fortran \
cmake openmpi python3 \
python-pip openssh \
Expand Down Expand Up @@ -88,35 +87,32 @@ You can now follow the appropriate instructions for your distribution.
</details>

<details>
<summary><h3>MacOS (x86 and Apple Silicon)</h3></summary>
<summary><h3>MacOS</h3></summary>

**Note:** macOS remains the most difficult platform to consistently compile MFC on.
If you run into issues, we suggest you try using Docker (instructions above).

- **MacOS v10.15 (Catalina) or newer [ZSH]** (Verify with `echo $SHELL`)
- **If you use [ZSH]** (Verify with `echo $SHELL`)

```console
$ touch ~/.zshrc
$ open ~/.zshrc
touch ~/.zshrc
open ~/.zshrc
```

- **Older than MacOS v10.15 (Catalina) [BASH]** (Verify with `echo $SHELL`)
- **If you use [BASH]** (Verify with `echo $SHELL`)

```console
$ touch ~/.bash_profile
$ open ~/.bash_profile
touch ~/.bash_profile
open ~/.bash_profile
```

An editor should open.
Please paste the following lines into it before saving the file.
If you wish to use a version of GNU's GCC other than 11, modify the first assignment.
If you wish to use a version of GNU's GCC other than 13, modify the first assignment.
These lines ensure that LLVM's Clang, and Apple's modified version of GCC, won't be used to compile MFC.
Further reading on `open-mpi` incompatibility with `clang`-based `gcc` on macOS: [here](https://stackoverflow.com/questions/27930481/how-to-build-openmpi-with-homebrew-and-gcc-4-9).
We do *not* support `clang` due to conflicts with our Silo dependency.
We do *not* support `clang` due to conflicts with the Silo dependency.

```console
# === MFC MPI Installation ===
export MFC_GCC_VER=11
export MFC_GCC_VER=13
export OMPI_MPICC=gcc-$MFC_GCC_VER
export OMPI_CXX=g++-$MFC_GCC_VER
export OMPI_FC=gfortran-$MFC_GCC_VER
Expand All @@ -126,11 +122,11 @@ export FC=gfortran-$MFC_GCC_VER
# === MFC MPI Installation ===
```

**Close the open editor and terminal window**. Open a **new terminal** window before executing the commands bellow.
**Close the open editor and terminal window**. Open a **new terminal** window before executing the commands below.

```console
$ brew install wget make python make cmake coreutils gcc@$MFC_GCC_VER
$ HOMEBREW_MAKE_JOBS=$(nproc) brew install --cc=gcc-$MFC_GCC_VER --verbose --build-from-source open-mpi
brew install wget make python make cmake coreutils gcc@$MFC_GCC_VER
HOMEBREW_MAKE_JOBS=$(nproc) brew install --cc=gcc-$MFC_GCC_VER --verbose --build-from-source open-mpi
```

They will download the dependencies MFC requires to build itself. `open-mpi` will be compiled from source, using the version of GCC we specified above with the environment variables `HOMEBREW_CC` and `HOMEBREW_CXX`.
Expand All @@ -149,22 +145,22 @@ First install Docker and Git:
- macOS: `brew install git docker` (requires [Homebrew](https://brew.sh/)).
- Other systems:
```console
$ sudo apt install git docker # Debian / Ubuntu via Aptitude
$ sudo pacman -S git docker # Arch Linux via Pacman
sudo apt install git docker # Debian / Ubuntu via Aptitude
sudo pacman -S git docker # Arch Linux via Pacman
```

Once Docker and Git are installed on your system, clone MFC with

```console
$ git clone https://github.com/MFlowCode/MFC
$ cd MFC
git clone https://github.com/MFlowCode/MFC
cd MFC
```

To fetch the prebuilt Docker image and enter an interactive bash session with the
recommended settings applied, run

```console
$ ./mfc.sh docker # If on \*nix/macOS
./mfc.sh docker # If on \*nix/macOS
.\mfc.bat docker # If on Windows
```

Expand Down Expand Up @@ -201,21 +197,21 @@ For a detailed list of options, arguments, and features, please refer to `./mfc.

Most first-time users will want to build MFC using 8 threads (or more!) with MPI support:
```console
$ ./mfc.sh build -j 8
./mfc.sh build -j 8
```

Examples:

- Build MFC using 8 threads with MPI and GPU acceleration: `$ ./mfc.sh build --gpu -j 8`.
- Build MFC using a single thread without MPI, GPU, and Debug support: `$ ./mfc.sh build --no-mpi`.
- Build MFC's `simulation` code in Debug mode with MPI and GPU support: `$ ./mfc.sh build --debug --gpu -t simulation`.
- Build MFC using 8 threads with MPI and GPU acceleration: `./mfc.sh build --gpu -j 8`.
- Build MFC using a single thread without MPI, GPU, and Debug support: `./mfc.sh build --no-mpi`.
- Build MFC's `simulation` code in Debug mode with MPI and GPU support: `./mfc.sh build --debug --gpu -t simulation`.

## Running the Test Suite

Run MFC's test suite with 8 threads:

```console
$ ./mfc.sh test -j 8
./mfc.sh test -j 8
```

Please refer to the [Testing](testing.md) document for more information.
Expand All @@ -225,7 +221,7 @@ Please refer to the [Testing](testing.md) document for more information.
MFC has example cases in the `examples` folder. You can run such a case interactively using 2 tasks by typing:

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -n 2
./mfc.sh run examples/2D_shockbubble/case.py -n 2
```

Please refer to the [Running](running.md) document for more information on `case.py` files and how to run them.
80 changes: 40 additions & 40 deletions docs/documentation/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Please refer to `./mfc.sh run -h` for a complete list of arguments and options,
To run all stages of MFC, that is [pre_process](https://github.com/MFlowCode/MFC/tree/master/src/pre_process/), [simulation](https://github.com/MFlowCode/MFC/tree/master/src/simulation/), and [post_process](https://github.com/MFlowCode/MFC/tree/master/src/post_process/) on the sample case [2D_shockbubble](https://github.com/MFlowCode/MFC/tree/master/examples/2D_shockbubble/),

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py
./mfc.sh run examples/2D_shockbubble/case.py
```

If you want to run a subset of the available stages, you can use the `-t` argument.
Expand All @@ -46,59 +46,59 @@ For example,
- Running [pre_process](https://github.com/MFlowCode/MFC/tree/master/src/pre_process/) with 2 cores:

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -t pre_process -n 2
./mfc.sh run examples/2D_shockbubble/case.py -t pre_process -n 2
```

- Running [simulation](https://github.com/MFlowCode/MFC/tree/master/src/simulation/) and [post_process](https://github.com/MFlowCode/MFC/tree/master/src/post_process/)
using 4 cores:

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4
./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4
```

## Batch Execution

The MFC detects which scheduler your system is using and handles the creation and execution of batch scripts.
The batch engine is requested via the `-e batch` option.
The number of nodes can be specified with the `-N` (i.e `--nodes`) option.
The number of nodes can be specified with the `-N` (i.e., `--nodes`) option.

We provide a list of (baked-in) submission batch scripts in the `toolchain/templates` folder.

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch -N 2 -n 4 -t simulation -c <computer name>
./mfc.sh run examples/2D_shockbubble/case.py -e batch -N 2 -n 4 -t simulation -c <computer name>
```

Other useful arguments include:

- `-# <job name>` to name your job. (i.e `--name`)
- `-@ sample@example.com` to receive emails from the scheduler. (i.e `--email`)
- `-w hh:mm:ss` to specify the job's maximum allowed walltime. (i.e `--walltime`)
- `-a <account name>` to identify the account to be charged for the job. (i.e `--account`)
- `-p <partition name>` to select the job's partition. (i.e `--partition`)
- `-# <job name>` to name your job. (i.e., `--name`)
- `-@ sample@example.com` to receive emails from the scheduler. (i.e., `--email`)
- `-w hh:mm:ss` to specify the job's maximum allowed walltime. (i.e., `--walltime`)
- `-a <account name>` to identify the account to be charged for the job. (i.e., `--account`)
- `-p <partition name>` to select the job's partition. (i.e., `--partition`)

As an example, one might request GPUs on a SLURM system using the following:

**Disclaimer**: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to
allocate resources. Therefore, the MFC constructs equivalent resource-sets in task and GPU count.
allocate resources. Therefore, the MFC constructs equivalent resource sets in the task and GPU count.

### Profiling with NVIDIA Nsight

MFC provides two different argument to facilitate profiling with NVIDIA Nsight.
**Please ensure that the used argument is placed at the end so that their respective flags can be appended.**
MFC provides two different arguments to facilitate profiling with NVIDIA Nsight.
**Please ensure the used argument is placed at the end so their respective flags can be appended.**
- Nsight Systems (Nsys): `./mfc.sh run ... --nsys [nsys flags]` allows one to visualize MFC's system-wide performance with [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems).
NSys is best for getting a general understanding of the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC.
NSys is best for understanding the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC.
When used, `--nsys` will run the simulation and generate `.nsys-rep` files in the case directory for all targets.
These files can then be imported into Nsight System's GUI, which can be downloaded [here](https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms). It is best to run case files with a few timesteps so that the report files remain small. Learn more about NVIDIA Nsight Systems [here](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
These files can then be imported into Nsight System's GUI, which can be downloaded [here](https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms). It is best to run case files with a few timesteps to keep the report files small. Learn more about NVIDIA Nsight Systems [here](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
- Nsight Compute (NCU): `./mfc.sh run ... --ncu [ncu flags]` allows one to conduct kernel-level profiling with [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute).
NCU provides profiling information for every subroutine called and is more detailed than NSys.
When used, `--ncu` will output profiling information for all subroutines, including elapsed clock cycles, memory used, and more after the simulation is run.
Please note that adding this argument will significantly slow down the simulation and should only be used on case files with a few timesteps.
Adding this argument will significantly slow the simulation and should only be used on case files with a few timesteps.
Learn more about NVIDIA Nsight Compute [here](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html).

### Restarting Cases

When running a simulation, MFC generates a `./restart_data` folder in the case directory that contains `lustre_*.dat` files that can be used to restart a simulation from saved timesteps.
This allows a user to run a simulation to some timestep $X$, then later continue it to run to another timestep $Y$, where $Y > X$.
This allows a user to simulate some timestep $X$, then continue it to run to another timestep $Y$, where $Y > X$.
The user can also choose to add new patches at the intermediate timestep.

If you want to restart a simulation,
Expand All @@ -108,29 +108,29 @@ If you want to restart a simulation,
- `t_step_stop` : $t_f$
- `t_step_save` : $SF$
in which $t_i$ is the starting time, $t_f$ is the final time, and $SF$ is the saving frequency time.
- Run pre-process and simulation on the case.
- Run `pre_process` and `simulation` on the case.
- `./mfc.sh run case.py -t pre_process simulation `
- As the simulation runs, it will create LUSTRE files for each saved timestep in `./restart_data`.
- When the simulation stops, choose any LUSTRE file as the restarting point (lustre_ $t_s$.dat)
- Create a new duplicate input file, (ex. `restart_case.py`), on which it should:
- As the simulation runs, it will create Lustre files for each saved timestep in `./restart_data`.
- When the simulation stops, choose any Lustre file as the restarting point (lustre_ $t_s$.dat)
- Create a new duplicate input file (e.g., `restart_case.py`), which should have:

1. For the Computational Domain Parameters
- Have the following removed BUT `m`, `n`, and `p`:
- All domaing/mesh information
- Have the following removed __except__ `m`, `n`, and `p`:
- All domain/mesh information
- `(xyz)_domain%beg`
- `(xyz)_domain%end`
- `stretch_(xyz)`
- `a_(xyz)`
- `(xyz)_a`
- `(xyz)_b`
- Have the following altered:
- `t_step_start` : $t_s$ # this is the point at which the simulation will restart
- `t_step_stop` : $t_{f2}$ # your new final simulation time, which can be the same as $t_f$
- `t_step_save` : ${SF}_2$ # if interested in changing the saving frequency
- Have the following ADDED:
- `old_ic` : 'T' # to specify that we have initial conditions from previous simulations
- `old_grid` : 'T' # to specify that we have a grid from previous simulations (maybe I do not need m, n, and p, then?)
- `t_step_old` : $t_i$ # this is the time step used as the `t_step_start` of the original `case.py` file
- Alter the following:
- `t_step_start` : $t_s$ (the point at which the simulation will restart)
- `t_step_stop` : $t_{f2}$ (new final simulation time, which can be the same as $t_f$)
- `t_step_save` : ${SF}_2$ (if interested in changing the saving frequency)
- Add the following:
- `old_ic` : 'T' (to specify that we have initial conditions from previous simulations)
- `old_grid` : 'T' (to specify that we have a grid from previous simulations)
- `t_step_old` : $t_i$ (the time step used as the `t_step_start` of the original `case.py` file)
2. For the Simulation Algorithm Parameters
- Substitute `num_patches` to reflect the number of ADDED patches in the `restart_case.py` file. If no patches are added, set `num_patches: 0`

Expand All @@ -145,32 +145,32 @@ in which $t_i$ is the starting time, $t_f$ is the final time, and $SF$ is the sa
4. For Fluid properties
- Keep information about the fluid properties

- Run pre-process and simulation on restart_case.py
- Run pre-process and simulation on `restart_case.py`
- `./mfc.sh run restart_case.py -t pre_process simulation `

- Run the post_process
- There are several ways to do this. Keep in mind that, regardless of the .py file used, the post_process command will generate output files in the [`t_step_start`, `t_step_stop`] range, with `t_step_save` as the spacing between files.
- One way is to set `t_step_stop` to the restarting point $t_s$ in `case.py`. Then, run the commands below. The first command will run on timesteps $[t_i, t_s]$. The second command will run on $[t_s, t_{f2}]$. Therefore, the whole range $[t_i, t_{f2}]$ will be post processed.

```console
$ ./mfc.sh run case.py -t post_process
$ ./mfc.sh run restart_case.py -t post_process
./mfc.sh run case.py -t post_process
./mfc.sh run restart_case.py -t post_process
```

We have provided an example `case.py` and `restart_case.py` in `/examples/1D_vacuum_restart/`. This simulation is a duplicate of the `1D_vacuum` case. It demonstrates stopping at timestep 7000, adding a new patch, and restarting the simulation. To test this code, run:
We have provided an example, `case.py` and `restart_case.py` in `/examples/1D_vacuum_restart/`. This simulation is a duplicate of the `1D_vacuum` case. It demonstrates stopping at timestep 7000, adding a new patch, and restarting the simulation. To test this code, run:

```console
$ ./mfc.sh run examples/1D_vacuum_restart/case.py -t pre_process simulation
$ ./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t pre_process simulation
$ ./mfc.sh run examples/1D_vacuum_restart/case.py -t post_process
$ ./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process
./mfc.sh run examples/1D_vacuum_restart/case.py -t pre_process simulation
./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t pre_process simulation
./mfc.sh run examples/1D_vacuum_restart/case.py -t post_process
./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process
```

### Example Runs

- Oak Ridge National Laboratory's [Summit](https://www.olcf.ornl.gov/summit/):

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch \
./mfc.sh run examples/2D_shockbubble/case.py -e batch \
-N 2 -n 4 -t simulation -a <redacted> -c summit
```
Loading

0 comments on commit 605ee69

Please sign in to comment.