Solution Freezing On 3rd Iteration Onwards When Running Unsteady Dual-Timestepping in Parallel

[j-signorelli]

Running unsteady simulations w/ dual-time stepping on many cores causes solution (and residuals) to freeze at 3rd outer iteration

After running unsteady simulations with dual-time stepping both in serial and parallel, I discovered an issue where using a relatively larger number of processes led to "frozen" residuals while running. An image of this is shown below:

These residual values remain throughout the remaining duration of the run. This does not occur when running in serial. Additionally, as expected, the solution does not change after this 3rd iteration. The serial run results do, and they match the data I was comparing against.

I use Intel MPI, where I set -Dcustom-mpi=true and set the following environment variables prior to calling meson:

export CC=$(which mpicc)
export CXX=$(which mpicxx)
export MPICC=$(which mpicc)
export MPICXX=$(which mpicxx)

I have noticed on a variety of other simulations I am running that the error trends with # of cores I am using.

Bug report checklist
Please make sure that you have followed the checklist below, many common problems can be solved by:

• Consulting the build instructions (https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/).
• Looking for similar problems on GitHub or CFD-Online (https://www.cfd-online.com/Forums/su2/).
• Updating to the newest version of SU2 (either develop, master, or the pre-built executables https://su2code.github.io/download.html).

Desktop (please complete the following information):

• OS: CentOS Linux 7 (Core)
• C++ compiler and version: g++ (GCC) 9.1.0
• MPI implementation and version: Intel MPI 19.0.9
• SU2 Version: v7.5.1 (also noticed on v7.5.0)

[j-signorelli]

Attached below is also my configuration file, if it is at all helpful.

laminar_config_unsteady.txt

[pcarruscag]

Can you add LINSOL to the screen output and post the one one that works and the one that freezes?

[j-signorelli]

Hi, thanks for the quick response. Attached is the serial below which works:
serial.txt

Parallel (ran on 15 cores this time, less than the # I used in the screenshot I first posted). This time it appears to occur on the 4th timestep):
parallel.txt

I stopped output manually ~.10 seconds

[pcarruscag]

Ok the last number is the residual of the linear solver, 0 means that fgmres is stagnating.
You can try lowering the CFL (or using CFL_ADAPT), or you can try running the case steady and then initialize, which should be better for the linear solver.

[j-signorelli]

This resolved it, thank you so much. Why does required CFL vary based on number of processes being used? Is data not being communicated every inner iteration between ranks?

[pcarruscag]

To parallelize the ILU factorization used to precondition the linear system we make it local to each rank. This tends to make it less effective as the number of ranks increases.