Skip to content

Conversation

c-p-schmidt
Copy link
Contributor

Description and Context

AMGnxn should be replaced by MueLu and Teko, see #754. This draft PR implements the first steps to achieve #754; however, there are still some further steps to take.
The BGS-AMG without dofsplit (splitting of the matrix only happens along fields [scatra-solid-thermo] and/or geometry [anode-electrolyte-cathode, i.e., a separation of the entire domain into non-overlapping subdomains] works fine besides some probably more powerful preconditioners that need to be chosen.
In contrast, the BGS-AMG with dofsplit (i.e., an additional split of the scatra dofs [c, phi]) works for the more complex scenario of solid-scatra-interaction (besides a too less powerful preconditioner), but it does not work for the pure scatra problem, when Teko runs into a segfault within Trilinos. The information provided to Teko for this process looks consistent, so I do not understand what is happening there.

@maxfirmbach: I hope you can give some further hints on what could be the origin of these problems.

And finally, some further information on the individual commits:

  • 1-2: Necessary changes to ScaTra in order to enable access to node and dof block maps necessary for the preconditioner
  • 3: Bug in compute_solver_parameters
    Hint: 1-3 should and will be merged separately
  • 4: Remove AMGnxn from input and some related code
  • 5: Move elch tests and framework to work with Teko (Besides BGS-AMG with dofsplit and AMG-BGS)
  • 6: Move sti tests and framework to work with Teko (Besides AMG-BGS)
  • 7: Move ssi tests and framework to work with Teko (Besides BGS-AMG with dofsplit)
  • 8: Move ssti tests and framework to work with Teko
  • 9: Temporary printers in Teko to verify the information passed to Teko before construction of the preconditioner
  • 10: Enable the BGS-AMG dofsplit for elch and ssi

So to sum up, each subframework or subfunctionality is moved in separate commits in commits 5-8 and 10.

Related Issues and Pull Requests

part of #754

Interested parties

@bwirthl @lkoeglmeier

@c-p-schmidt c-p-schmidt requested a review from maxfirmbach July 7, 2025 13:27
@c-p-schmidt c-p-schmidt self-assigned this Jul 7, 2025
@c-p-schmidt c-p-schmidt marked this pull request as draft July 7, 2025 13:27
@c-p-schmidt
Copy link
Contributor Author

And here is an overview of the tests that are affected:

Finished besides changing the results ✅

  • elch_3D_tet4_s2i_butlervolmer_BGS-AMG_3x3.4C.yaml-p3 ...............................................................................................................................***Failed 36.22 sec
    -> Solver solves nicely, but result check fails which is expected
  • elch_3D_tet4_s2i_butlervolmer_mortar_standard_BGS-AMG_3x3.4C.yaml-p3 ...............................................................................................................***Failed 36.40 sec
    -> Solver solves nicely, but result check fails which is expected
  • elch_3D_tet4_s2i_butlervolmer_mortar_standard_redist_BGS-AMG_3x3.4C.yaml-p3 ........................................................................................................***Failed 30.64 sec
    -> Solver solves nicely, but result check fails which is expected
  • elch_3D_tet4_s2i_butlervolmer_plating_monolithic_reg_trig_BGS-AMG_4x4.4C.yaml-p3 ...................................................................................................***Failed 103.17 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_BGS-AMG_2x2.4C.yaml-p3 .....................................................................................................***Failed 53.64 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_condensed_BGS-AMG_2x2.4C.yaml-p3 ...........................................................................................***Failed 51.41 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_BGS-AMG_4x4.4C.yaml-p3 .....................................................................................................***Failed 80.97 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_condensed_BGS-AMG_4x4.4C.yaml-p3 ...........................................................................................***Failed 75.63 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_condensed_mortar_standard_BGS-AMG_2x2.4C.yaml-p3 ...........................................................................***Failed 36.39 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_mortar_standard_BGS-AMG_2x2.4C.yaml-p3 .....................................................................................***Failed 38.75 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_hex8_elch_s2i_butlervolmerpeltier_adiabatic_condensed_mortar_standard_BGS-AMG_4x4.4C.yaml-p3 ......................................................................***Failed 44.90 sec
    -> Solver solves nicely, but result check fails which is expected
  • sti_mono_3D_tet4_hex8_elch_s2i_butlervolmerpeltier_adiabatic_mortar_standard_BGS-AMG_4x4.4C.yaml-p3 ................................................................................***Failed 48.59 sec
    -> Solver solves nicely, but result check fails which is expected

Preconditioner too bad

  • ssi_mono_3D_hex8_elch_s2i_butlervolmer_grain_boundary_meshtying_BGS-AMG_5x5.4C.yaml-p2 .............................................................................................***Failed 4.22 sec
Iter  140, [ 1] :    1.369391e-02
Iter  145, [ 1] :    9.107705e-04
Iter  150, [ 1] :    2.866800e-05
Iter  155, [ 1] :    2.058302e-06
Iter  160, [ 1] :    1.618146e-06
Iter  165, [ 1] :    1.317104e-07
Iter  170, [ 1] :    6.158300e-08
Iter  175, [ 1] :    2.994942e-08
Iter  180, [ 1] :    1.332235e-08
Iter  185, [ 1] :    1.132661e-08
Iter  187, [ 1] :    7.252641e-09
Belos::PseudoBlockGmresSolMgr::solve(): Warning! Solver has experienced a loss of accuracy!
=========================================================================
PROC 0 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad

  • ssi_mono_3D_hex8_elch_s2i_butlervolmercapacity_BGS-AMG_4x4.4C.yaml-p2 ..............................................................................................................***Failed 3.03 sec
Iter  885, [ 1] :    1.202636e-01
Iter  890, [ 1] :    1.197272e-01
Iter  895, [ 1] :    1.191979e-01
Iter  900, [ 1] :    1.186756e-01
Iter  905, [ 1] :    3.360798e+00
Iter  910, [ 1] :    3.210767e+00
Iter  915, [ 1] :    3.150261e+00
Iter  920, [ 1] :    3.090965e+00
Iter  925, [ 1] :    3.001797e+00
Iter  930, [ 1] :    2.973915e+00
Iter  935, [ 1] :    2.947395e+00
Iter  940, [ 1] :    2.913978e+00
Iter  945, [ 1] :    2.861224e+00
Iter  950, [ 1] :    2.810215e+00
Iter  955, [ 1] :    2.741332e+00
Iter  960, [ 1] :    2.702915e+00
Iter  965, [ 1] :    2.684234e+00
Iter  970, [ 1] :    2.548493e+00
Iter  975, [ 1] :    2.151351e+00
Iter  980, [ 1] :    2.017465e+00
Iter  985, [ 1] :    1.765070e+00
Iter  990, [ 1] :    8.722739e-01
Iter  995, [ 1] :    6.895343e-01
Iter 1000, [ 1] :    2.255960e-01

=========================================================================
PROC 1 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad

  • ssi_mono_3D_tet4_elch_s2i_butlervolmer_manifold_BGS-AMG_3x3.4C.yaml-p3 .............................................................................................................***Failed 4.02 sec
Iter  865, [ 1] :    9.009990e-03
Iter  870, [ 1] :    9.007772e-03
Iter  875, [ 1] :    9.006525e-03
Iter  880, [ 1] :    9.004372e-03
Iter  885, [ 1] :    9.003852e-03
Iter  890, [ 1] :    8.999490e-03
Iter  895, [ 1] :    8.994626e-03
Iter  900, [ 1] :    8.993271e-03
Iter  905, [ 1] :    8.993271e-03
Iter  910, [ 1] :    8.993271e-03
Iter  915, [ 1] :    8.993271e-03
Iter  920, [ 1] :    8.993271e-03
Iter  925, [ 1] :    8.993271e-03
Iter  930, [ 1] :    8.993271e-03
Iter  935, [ 1] :    8.993271e-03
Iter  940, [ 1] :    8.993271e-03
Iter  945, [ 1] :    8.993271e-03
Iter  950, [ 1] :    8.993271e-03
Iter  955, [ 1] :    8.993271e-03
Iter  960, [ 1] :    8.993271e-03
Iter  965, [ 1] :    8.993271e-03
Iter  970, [ 1] :    8.993271e-03
Iter  975, [ 1] :    8.993271e-03
Iter  980, [ 1] :    8.993271e-03
Iter  985, [ 1] :    8.993271e-03
Iter  990, [ 1] :    8.993271e-03
Iter  995, [ 1] :    8.993271e-03
Iter 1000, [ 1] :    8.993271e-03
=========================================================================
PROC 1 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad

  • ssi_mono_3D_tet4_elch_s2i_butlervolmer_manifold_BGS-AMG_6x6.4C.yaml-p3 .............................................................................................................***Failed 0.98 sec
Iter  105, [ 1] :    1.866550e-04
Iter  110, [ 1] :    2.320085e-06
Iter  115, [ 1] :    1.532587e-07
Iter  119, [ 1] :    6.792557e-09
Belos::PseudoBlockGmresSolMgr::solve(): Warning! Solver has experienced a loss of accuracy!
=========================================================================
PROC 0 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad

  • ssti_mono_3D_6hex8_elch_s2i_butlervolmerthermo_BGS-AMG_3x3.4C.yaml-p2 ..............................................................................................................***Failed 0.89 sec
Iter  925, [ 1] :    1.836439e-01
Iter  930, [ 1] :    1.019956e-01
Iter  935, [ 1] :    9.533681e-02
Iter  940, [ 1] :    9.278695e-02
Iter  945, [ 1] :    9.019854e-02
Iter  950, [ 1] :    8.938341e-02
Iter  955, [ 1] :    8.677166e-02
Iter  960, [ 1] :    8.111195e-02
Iter  965, [ 1] :    7.566726e-02
Iter  970, [ 1] :    6.452227e-02
Iter  975, [ 1] :    3.363782e-02
Iter  980, [ 1] :    3.265104e-02
Iter  985, [ 1] :    3.239676e-02
Iter  990, [ 1] :    3.214833e-02
Iter  995, [ 1] :    3.190552e-02
Iter 1000, [ 1] :    3.166814e-02
=========================================================================
PROC 0 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad

  • ssti_mono_3D_tet4_batt_with_anode_plate_and_current_collectors_elch_s2i_butlervolmer_BGS_AMG_11x11.4C.yaml-p3 ......................................................................***Failed 0.90 sec
Iter   10, [ 1] :    6.436398e-03
Iter   15, [ 1] :    2.486579e-06
Iter   17, [ 1] :    7.672161e-09
Belos::PseudoBlockGmresSolMgr::solve(): Warning! Solver has experienced a loss of accuracy!
=========================================================================
PROC 2 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad (This is only for the scatra field during init pot calc. If we allow to ignore this error in GMRES it also starts with the first TS, where all contributions are solved for but does not properly solve then.

  • ssi_mono_3D_tet4_elch_s2i_butlervolmer_manifold_BGS-AMG_11x11.4C.yaml-p3 ...........................................................................................................***Failed 0.88 sec
Iter   75, [ 1] :    5.272179e-04
Iter   80, [ 1] :    1.027570e-04
Iter   85, [ 1] :    1.949314e-05
Iter   90, [ 1] :    4.298326e-06
Iter   95, [ 1] :    1.500318e-06
Iter   98, [ 1] :    9.234936e-07
Belos::PseudoBlockGmresSolMgr::solve(): Warning! Solver has experienced a loss of accuracy!
=========================================================================
PROC 0 ERROR in /home/schmidt/workspace/FourC/src/core/linear_solver/src/method/4C_linear_solver_method_iterative.cpp, line 174:
Core::LinearSolver::BelosSolver: Iterative solver did not converge.

-> Preconditioner too bad (This is only for the scatra field during init pot calc. If we allow to ignore this error in GMRES it also starts with the first TS, where all contributions are solved for but does not properly solve then.

Not yet working

  • elch_3D_tet4_s2i_butlervolmer_BGS-AMG_4x4.4C.yaml-p3 ...............................................................................................................................***Failed 0.97 sec
    -> BGS-AMG with dofsplit, runs into a segfault upon building Teko despite (all) info that is handed to Teko looks good in my opinion. Also the BGS-AMG with dofsplit works for ssi_mono_3D_tet4_elch_s2i_butlervolmer_manifold_BGS-AMG_11x11.4C.yaml-p3 besides a to less powerful preconditioner

Not yet tackled

  • elch_3D_tet4_s2i_butlervolmer_AMG-BGS_3x3.4C.yaml-p3 ...............................................................................................................................***Failed 1.00 sec
    -> AMG-BGS not yet migrated, we need to talk about that on how this is possible to achieve
  • elch_3D_tet4_s2i_butlervolmer_AMG-BGS_3x3.4C.yaml-p3 ...............................................................................................................................***Failed 1.00 sec
    -> AMG-BGS not yet migrated, we need to talk about that on how this is possible to achieve

beloslist.set("SOLVER_XML_FILE", xmlfile->string());

// required for adaptive linear solver tolerance
beloslist.set("Convergence Tolerance", inparams.get<double>("AZTOL"));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be solved differently.

what they actually contain and distinguish from the node_block maps
that need to be specified
and related implementation.
is working with Teko.
- remove old now unused xml files
- add new block preconditioner xml files for usage with Teko

The BGS-AMG for elch works with the following exception:
- the dof-split is not working
- AMG-BGS is not yet working
- test case results not yet adapted
- AMG-BGS not yet migrated
however, several not yet working properly
- BGS-AMG with dofsplit not yet tackled
There is still a segfault, I don't know why yet
Info that is set looks good, so I don't know what's the problem there
and the parameter list shit as well here and in elch
@c-p-schmidt c-p-schmidt force-pushed the move-remaining-scatra-to-teko branch from d5a3b99 to 32d4683 Compare August 18, 2025 09:31
@c-p-schmidt
Copy link
Contributor Author

After the session with @maxfirmbach last week and some further work, everything besides the dofsplit works. Thanks again!

I'll separate the individual changes to separate PR's and will close this one once everything is merged in smaller chunks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant