FDS UG: Add UGLMAT HYPRE discussion.

Manuals/FDS_User_Guide/FDS_User_Guide.tex

@@ -9245,9 +9245,12 @@ \subsection{Optional Pressure Solvers}
 
 The \ct{GLMAT} solver is used mainly for testing and diagnostics. Because it solves the pressure over the entire computational domain, it is limited to about 1~million cells total unless your computer has an extraordinary amount of RAM.
 
-\item \ct{SOLVER='UGLMAT'} This solver is for non-overlapping, non-stretched meshes at the same refinement level covering a single connected domain, i.e., a large rectangular box. Like the solver \ct{'ULMAT'}, the unknown values of the pressure live only in gas-phase cells, allowing for the normal components of velocity at a solid surface to be computed exactly (no velocity penetration error). In addition, this solver ensures that the normal component of velocity matches perfectly at mesh boundaries. This solver is limited to about 1~million grid cells, and (3) it is the most CPU-intensive of the alternative solvers.
+\item \ct{SOLVER='UGLMAT'} This solver is for non-overlapping, non-stretched meshes at the same refinement level. Like the solver \ct{'ULMAT'}, the unknown values of the pressure live only in gas-phase cells, allowing for the normal components of velocity at a solid surface to be computed exactly (no velocity penetration error). In addition, this solver ensures that the normal component of velocity matches perfectly at mesh boundaries. It is limited to about 1~million grid cells, and (3) it is the most CPU-intensive of the alternative solvers.
+
+\item \ct{SOLVER='UGLMAT HYPRE'} Performs the same global solution as \ct{'UGLMAT'} calling the HYPRE algebraic multigrid (AMG) preconditioned conjugate gradient (PCG) solver. It is also CPU-intensive, but does not have the global mesh size limitation of the previous method.
+
 \end{itemize}
-Because these alternative methods are based on the Intel\textsuperscript{\textregistered} MKL parallel LU decomposition solver, they require computation and storage of global factorization matrices. Depending on the size of the problem, these operations can be expensive in terms of both CPU time and memory. It has been found suitable for problems with up to a million cells on a multi-core workstation with 64~GB of RAM.
+Because the \ct{'ULMAT','GLMAT','UGLMAT'} alternative methods are based on the Intel\textsuperscript{\textregistered} MKL parallel LU decomposition solver, they require computation and storage of global factorization matrices. Depending on the size of the problem, these operations can be expensive in terms of both CPU time and memory. It has been found suitable for problems with up to a million cells on a multi-core workstation with 64~GB of RAM.
 
 
 \begin{landscape}
@@ -9269,7 +9272,8 @@ \subsection{Optional Pressure Solvers}
 GLMAT             & $\times$    & $\times$      & \checkmark & \checkmark & exact     & iterative  & iterative \\
 \rowcolor{lavender}
 UGLMAT            & $\times$    & $\times$      & \checkmark & \checkmark & exact     & exact      & iterative \\
-% PFFT$^7$ (future) & $\times$    & 1 direction   & \checkmark & \checkmark & exact     & iterative  & iterative \\
+UGLMAT HYPRE$^7$  & $\times$    & $\times$      & \checkmark & \checkmark & exact     & exact      & iterative \\
+% PFFT$^8$ (future) & $\times$    & 1 direction   & \checkmark & \checkmark & exact     & iterative  & iterative \\
 \bottomrule
 \end{tabular}
 \begin{tablenotes}
@@ -9282,7 +9286,8 @@ \subsection{Optional Pressure Solvers}
         \item[$^4$] ``Velocity @ Solid'' refers to errors in the normal component of velocity at solid boundaries, so-called ``penetration'' errors that are present in \emph{immersed boundary methods} and lead to mass conservation errors.  These errors are particularly important to control for tightly sealed pressure zones. For inexact solvers, this error may be controlled using the parameter \ct{VELOCITY_TOLERANCE}. Mass conservation errors are proportional to the velocity tolerance.  Solvers listed as ``exact'' are \emph{unstructured} solvers.  ``Unstructured'' means the solid boundaries (e.g., \ct{OBST} boundaries) are treated as domain boundaries for the pressure Poisson equation.  Unstructured solvers are the opposite of \emph{immersed boundary methods}.
         \item[$^5$] ``Inseparability'' refers to handling the error compared to the solution of the \emph{inseparable} Poisson equation incurred by lagging the pressure in the baroclinic term.  This error may be controlled using the parameter \ct{PRESSURE_TOLERANCE}.
         \item[$^6$] ULMAT HYPRE uses the HYPRE library developed at Lawrence Livermore National Laboratories (LLNL).  Use this variant of ULMAT if the Intel MKL library is not available \emph{or} if MKL runs out of memory or is too slow due to memory issues with large mesh blocks.  Our implementation of HYPRE uses an algebraic multigrid (AMG) preconditioned conjugate gradient (PCG) method, which does not require excessive amounts of memory.
-        % \item[$^7$] PFFT stands for Parallel Fast Fourier Transform.  This solver is only possible for block structured domains.  The domain decomposition for the pressure solver differs from the transport solver.  It uses pencil-shaped domains that span each direction and thus perform FFT solves in each direction for the whole domain.  Data is transferred in each stage of the solution to the orthogonal directions.  This solver is \emph{global} but \emph{structured}.  So, mesh to mesh errors are eliminated but immersed boundary errors are not.
+        \item[$^7$] Use this variant of UGLMAT for cases where the total number of cells is greater than 1 million.
+        % \item[$^8$] PFFT stands for Parallel Fast Fourier Transform.  This solver is only possible for block structured domains.  The domain decomposition for the pressure solver differs from the transport solver.  It uses pencil-shaped domains that span each direction and thus perform FFT solves in each direction for the whole domain.  Data is transferred in each stage of the solution to the orthogonal directions.  This solver is \emph{global} but \emph{structured}.  So, mesh to mesh errors are eliminated but immersed boundary errors are not.
     \end{itemize}
 \end{tablenotes}
 \end{threeparttable}
@@ -9293,7 +9298,7 @@ \subsection{Optional Pressure Solvers}
 \subsection{Example Case: \ct{Pressure_Solver/duct_flow}}
 \label{duct_flow}
 
-To demonstrate how to improve the accuracy of the pressure solver, consider the flow of air through a square duct that crosses several meshes. In the sample input file, \ct{duct_flow.fds}, air is pushed through a 1 m$^2$ duct at 1~m/s. With no thermal expansion, the volume flow into the duct ought to equal the volume flow out of the duct. Figure~\ref{duct_flow_fig} displays the computed inflow and outflow as a function of time and the number of pressure iterations required. The default pressure solution strategy uses block-wise direct FFT solves combined with a direct forcing immersed boundary method to model flow obstructions.  Hence, there are two sources of pressure error: one at a mesh interface and one at a solid boundary.  The default pressure iteration scheme tries to drive both sources of error to the specified (or default) velocity tolerance. In the \ct{duct_flow} case, the \ct{VELOCITY_TOLERANCE} has been set to 0.001~m/s with \ct{MAX_PRESSURE_ITERATIONS} set to 1000 and the grid cell size is 0.2~m.  For the default scheme (FFT + IBM), the outflow does not match the inflow exactly because of inaccuracies at the solid and mesh boundaries.  We have also included the results using the \ct{'UGLMAT'} solver with an unstructured pressure matrix, which allows the flow solver to enforce the impermeability condition (zero normal component of velocity) at a solid boundary.
+To demonstrate how to improve the accuracy of the pressure solver, consider the flow of air through a square duct that crosses several meshes. In the sample input file, \ct{duct_flow.fds}, air is pushed through a 1 m$^2$ duct at 1~m/s. With no thermal expansion, the volume flow into the duct ought to equal the volume flow out of the duct. Figure~\ref{duct_flow_fig} displays the computed inflow and outflow as a function of time and the number of pressure iterations required. The default pressure solution strategy uses block-wise direct FFT solves combined with a direct forcing immersed boundary method to model flow obstructions.  Hence, there are two sources of pressure error: one at a mesh interface and one at a solid boundary.  The default pressure iteration scheme tries to drive both sources of error to the specified (or default) velocity tolerance. In the \ct{duct_flow} case, the \ct{VELOCITY_TOLERANCE} has been set to 0.001~m/s with \ct{MAX_PRESSURE_ITERATIONS} set to 1000 and the grid cell size is 0.2~m.  For the default scheme (FFT + IBM), the outflow does not match the inflow exactly because of inaccuracies at the solid and mesh boundaries.  We have also included the results using the \ct{'UGLMAT','UGLMAT HYPRE'} solvers with an unstructured pressure matrix, which allows the flow solver to enforce the impermeability condition (zero normal component of velocity) at a solid boundary.
 \begin{lstlisting}
 &PRES SOLVER='UGLMAT' /
 \end{lstlisting}
@@ -9312,7 +9317,7 @@ \subsection{Example Case: \ct{Pressure_Solver/duct_flow}}
 \subsection{Example Case: \ct{Pressure_Solver/dancing_eddies}}
 \label{dancing_eddies}
 
-In this example, air is pushed through a 30~cm long, two-dimensional channel at 0.5~m/s. A plate obstruction normal to the flow creates a Karman vortex street. The computational domain is divided into 4 meshes. Three simulations are performed: one in which the \ct{VELOCITY_TOLERANCE} is set to the default value, one in which it is set to a relatively small value, and one in which the \ct{SOLVER='UGLMAT'} is used. Figure~\ref{dancing_eddies_figure} shows the downstream pressure histories for these cases compared to a simulation that uses only one mesh. The case with the tighter tolerance produces a better match to the single mesh solution, but at a higher computational cost---about a factor of 2.5 .  The \ct{SOLVER='UGLMAT'} case takes just a little longer than the default---by a factor of 1.25---but the error is machine precision.  These cases used MPI for domain decomposition, but only a single OpenMP thread.  Note, however, that the \ct{'UGLMAT'} solver is threaded, whereas currently the \ct{'FFT'} solver (FDS default) is not.
+In this example, air is pushed through a 30~cm long, two-dimensional channel at 0.5~m/s. A plate obstruction normal to the flow creates a Karman vortex street. The computational domain is divided into 4 meshes. Six simulations are performed: one in which the \ct{VELOCITY_TOLERANCE} is set to the default value, one in which it is set to a relatively small value, two using \ct{'ULMAT'} and \ct{'ULMAT HYPRE'}, and two where \ct{'UGLMAT'} and \ct{'UGLMAT HYPRE'} are used. Figure~\ref{dancing_eddies_figure} shows the downstream pressure histories for these cases compared to a simulation that uses only one mesh. The case with the tighter tolerance produces a better match to the single mesh solution, but at a higher computational cost---about a factor of 2.5 .  The \ct{SOLVER='UGLMAT'} case takes just a little longer than the default---by a factor of 1.25---but the error is machine precision.  These cases used MPI for domain decomposition, but only a single OpenMP thread.  Note, however, that the \ct{'UGLMAT'} solver is threaded, whereas currently the \ct{'FFT'} solver (FDS default) is not.
 
 \begin{figure}[ht]
 \begin{center}