Skip to content

Conversation

maxfirmbach
Copy link
Contributor

@maxfirmbach maxfirmbach commented Apr 2, 2025

Description and Context

This PR mainly consists of the first changes moving from Isorropia to Zoltan2 and for now mostly acts to view code changes and observe failing tests.

Current problems:

Related Issues and Pull Requests

@maxfirmbach maxfirmbach added status: in progress Issues that are currently being worked on taskforce: tpetra Issues related to the migration from Epetra to Tpetra team: discretization labels Apr 2, 2025
@maxfirmbach maxfirmbach added this to the Migration to Tpetra milestone Apr 2, 2025
@maxfirmbach maxfirmbach self-assigned this Apr 2, 2025
@maxfirmbach maxfirmbach marked this pull request as draft April 2, 2025 10:04
@isteinbrecher
Copy link
Contributor

isteinbrecher commented Apr 2, 2025

I took the liberty of collecting all failing tests:


 20 - beam3eb_backweuler_browndyn_singlefil.4C.yaml-p3 (Failed)
 31 - beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_first.4C.yaml-p2_for_beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_second.4C.yaml-p3-restart_step_1 (Failed)
 32 - beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_second.4C.yaml-p3_from_beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_first.4C.yaml-p2-restart_step_1 (Not Run)
 47 - beam3r_herm2line3_backweuler_browndyn_crosslinking_beam3rline2_belloffrate.4C.yaml-p2 (Failed)
 52 - beam3r_herm2line3_backweuler_browndyn_randcrosslinking_beam3rline2.4C.yaml-p2 (Failed)
 53 - beam3r_herm2line3_backweuler_browndyn_randcrosslinking_beam3rline2_twobonds.4C.yaml-p3 (Failed)
 56 - beam3r_herm2line3_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 57 - beam3r_herm2line3_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 186 - beam3r_line2_backweuler_browndyn_initialcrosslinking_beam3rline2_largesys.4C.yaml-p3 (Failed)
 187 - beam3r_line2_backweuler_browndyn_largesys.4C.yaml-p3 (Failed)
 194 - beam3r_line2_backweuler_browndyn_randcrosslinking_beam3rline2_largesys.4C.yaml-p3 (Failed)
 197 - beam3r_line2_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 198 - beam3r_line2_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 228 - beam3r_rigidsphere_activecrosslinking_truss_browndyn_truss_catchslipbond.4C.yaml-p3 (Failed)
 229 - beam3r_rigidsphere_activecrosslinking_truss_browndyn_truss_forcedependetoffrate.4C.yaml-p3 (Failed)
 255 - beam3wk_tanline3_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 256 - beam3wk_tanline3_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 327 - contact2D_parredist.4C.yaml-p2 (Failed)
 328 - contact2D_parredist_mineleproc.4C.yaml-p3 (Failed)
 471 - elch_1D_line2_multiscale_butlervolmer_electrolyte.4C.yaml-p3 (Failed)
 472 - elch_1D_line2_multiscale_butlervolmer_electrolyte.4C.yaml-p3-restart_from_same_input (Not Run)
 473 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro.4C.yaml-p3 (Failed)
 474 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro.4C.yaml-p3-restart_from_same_input (Not Run)
 475 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro_aitken.4C.yaml-p3 (Failed)
 476 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro_aitken_dofsplit.4C.yaml-p3 (Failed)
 477 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_microtomacro.4C.yaml-p3 (Failed)
 539 - elch_3D_hex8_line2_scl_2dof_BGS_AMG_2x2.4C.yaml-p2-csv_comparison-xxx_micro_macro_coupling_coords.csv (Failed)
 547 - elch_3D_tet4_s2i_butlervolmer_AMG_pointbased.4C.yaml-p3 (Failed)
 548 - elch_3D_tet4_s2i_butlervolmer_AMG_vectorbased.4C.yaml-p3 (Failed)
 549 - elch_3D_tet4_s2i_butlervolmer_AMG-BGS_3x3.4C.yaml-p3 (Failed)
 550 - elch_3D_tet4_s2i_butlervolmer_BGS-AMG_3x3.4C.yaml-p3 (Failed)
 562 - elch_3D_tet4_s2i_butlervolmer_mortar_standard_redist.4C.yaml-p3 (Failed)
 563 - elch_3D_tet4_s2i_butlervolmer_mortar_standard_redist_BGS-AMG_3x3.4C.yaml-p3 (Failed)
 576 - elch_gaussian_hill_pbc_genalpha.4C.yaml-p2 (Failed)
 577 - elch_gaussian_hill_pbc_genalpha.4C.yaml-p2-restart_from_same_input (Not Run)
 579 - elch_gaussian_hill_pbc_NP_genalpha_calcInitDt_calcInitPot.4C.yaml-p2 (Failed)
 581 - elch_Kwok_Wu_BDF2.4C.yaml-p2 (Failed)
 582 - elch_Kwok_Wu_BDF2.4C.yaml-p2-restart_from_same_input (Not Run)
 602 - elch_pulseplating_hex27_moving_boundary.4C.yaml-p3 (Failed)
 603 - elch_pulseplating_hex27_moving_boundary.4C.yaml-p3-restart_from_same_input (Not Run)
 604 - elch_RCE_rotsymm_pbc.4C.yaml-p2 (Failed)
 709 - f3_bfs_xwall_8x8x8_bl1_turbinflow.4C.yaml-p3 (Failed)
 710 - f3_cha_8x8x8_recongradl2.4C.yaml-p2 (Failed)
 711 - f3_cha_8x8x8_recongradspr.4C.yaml-p2 (Failed)
 712 - f3_cha_xwall_6x8x6.4C.yaml-p3 (Failed)
 714 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp.4C.yaml-p2 (Failed)
 715 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp.4C.yaml-p2-restart_from_same_input (Not Run)
 716 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp_ale_mfs.4C.yaml-p2 (Failed)
 718 - f3_cha_xwall_6x8x6_gradres_p_bs_ramp_ale.4C.yaml-p2 (Failed)
 741 - f3_channel_meshtying_DCmaster.4C.yaml-p2 (Failed)
 742 - f3_channel_meshtying_DCmaster.4C.yaml-p2-restart_from_same_input (Not Run)
 743 - f3_channel_mixedhybrid.4C.yaml-p3 (Failed)
 744 - f3_channel_mixedhybrid.4C.yaml-p3-restart_from_same_input (Not Run)
 751 - f3_drt_genalpha_initialfield_volforce_pbc.4C.yaml-p2 (Failed)
 752 - f3_drt_impl_initialfield_volforce_pbc.4C.yaml-p2 (Failed)
 756 - f3_nurbs27_AfGenAlpha.4C.yaml-p3 (Failed)
 757 - f3_periodic_hill_w_forcing_24x12x12.4C.yaml-p3 (Failed)
 760 - f3_shear_flow_linear_ramp_periodic_fix_pressure_new_ost.4C.yaml-p3 (Failed)
 762 - f3_turbinflow_doublechan.4C.yaml-p2 (Failed)
 763 - f3_turbinflow_doublechan.4C.yaml-p2-restart_from_same_input (Not Run)
 764 - f3_turbinflow_doublechan_transferdbc.4C.yaml-p2 (Failed)
 776 - f3_womersley.4C.yaml-p2 (Failed)
 777 - f3_womersley.4C.yaml-p2-restart_from_same_input (Not Run)
 806 - fsci_simp_salz_syst.4C.yaml-p3 (Failed)
 824 - fsi_dc_mono_ss_ada_euler_none.4C.yaml-p2 (Failed)
 839 - fsi_dc3D_part_ait_ga_ost_xwall.4C.yaml-p3 (Failed)
 874 - fsi_pw_mono_fs_ga_ga_contact.4C.yaml-p2 (Failed)
 925 - levelset_gaussian_hill_pbc.4C.yaml-p2 (Failed)
 926 - levelset_gaussian_hill_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 945 - loma_turbmodel_test_dynsmag.4C.yaml-p2 (Failed)
 946 - loma_turbmodel_test_dynsmag.4C.yaml-p3 (Failed)
 1402 - poro_3D_hex8_nopenetration.4C.yaml-p2 (Failed)
 1566 - scatra_1D_line2_multiscale_constperm_macro.4C.yaml-p3 (Failed)
 1567 - scatra_1D_line2_multiscale_constperm_macro.4C.yaml-p3-restart_from_same_input (Not Run)
 1568 - scatra_1D_line2_multiscale_constperm_macro_partitioned_macrotomicro.4C.yaml-p3 (Failed)
 1569 - scatra_1D_line2_multiscale_constperm_macro_partitioned_macrotomicro.4C.yaml-p3-restart_from_same_input (Not Run)
 1570 - scatra_1D_line2_multiscale_constperm_macro_partitioned_microtomacro.4C.yaml-p3 (Failed)
 1612 - scatra_3D_tet4_hex8_s2i_constperm_mortar_standard.4C.yaml-p3 (Failed)
 1613 - scatra_3D_tet4_hex8_s2i_constperm_mortar_standard.4C.yaml-p3-restart_from_same_input (Not Run)
 1627 - scatra_gaussian_hill_pbc.4C.yaml-p2 (Failed)
 1628 - scatra_gaussian_hill_pbc_gena_cons.4C.yaml-p2 (Failed)
 1629 - scatra_gaussian_hill_pbc_genalpha.4C.yaml-p2 (Failed)
 1630 - scatra_gaussian_hill_pbc_genalpha.4C.yaml-p2-restart_from_same_input (Not Run)
 1641 - scatra_gaussian_hill_pbc_reaBioFilm_noStab.4C.yaml-p2 (Failed)
 1642 - scatra_gaussian_hill_pbc_reaBioFilm_THZ.4C.yaml-p2 (Failed)
 1694 - scatra_turbmodel_test_dynsmag.4C.yaml-p3 (Failed)
 1699 - scatra_turbmodel_test_mfs_par.4C.yaml-p3 (Failed)
 1700 - scatra_turbmodel_test_mfs_par.4C.yaml-p3-restart_from_same_input (Not Run)
 1701 - scatra_turbmodel_test_vreman.4C.yaml-p3 (Failed)
 1739 - sohex8_multiscale_macro.4C.yaml-p2 (Failed)
 1740 - sohex8_multiscale_macro.4C.yaml-p2-restart_from_same_input (Not Run)
 1741 - sohex8_multiscale_macro_2micro.4C.yaml-p2 (Failed)
 1742 - sohex8_multiscale_macro_2micro.4C.yaml-p2-restart_from_same_input (Not Run)
 2076 - ssi_1wc_membrane_active_strain_scatrafromrestart_vol_nonmatching.4C.yaml-p2_from_ssi_1wc_membrane_active_strain_scatra_only.4C.yaml-p2-restart_step_0 (Timeout)
 2185 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_AMG_pointbased.4C.yaml-p3 (Failed)
 2187 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_BGS-AMG_2x2.4C.yaml-p3 (Failed)
 2188 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_BGS-AMG_2x2.4C.yaml-p3-restart_from_same_input (Not Run)
 2189 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_condensed_BGS-AMG_2x2.4C.yaml-p3 (Failed)
 2190 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_condensed_BGS-AMG_2x2.4C.yaml-p3-restart_from_same_input (Not Run)
 2194 - sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_AMG-BGS_4x4.4C.yaml-p3 (Failed)
 2286 - tsi_heatflux_monolithic_pbc.4C.yaml-p2 (Failed)
 2287 - tsi_heatflux_monolithic_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 2335 - w1nurbs9_pbc.4C.yaml-p2 (Failed)
 2336 - w1nurbs9_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 2504 - xfluid_kimmoin_stat_24x24x1_wedge15_levelset.4C.yaml-p3 (Timeout)
 2539 - xfluid_taylor_couette_NavSlip_eps1.0_25x25_krylov_levelset_altgeogeneration.4C.yaml-p2 (Failed)
 1739 - sohex8_multiscale_macro.4C.yaml-p2 (Failed)
 2616 - beam3eb_static_beam_to_solid_volume_meshtying_monolithic_partitioning-vtk-beam.4C.yaml-p3 (Failed)
 2617 - beam3eb_static_beam_to_solid_volume_meshtying_monolithic_partitioning-vtk-structure.4C.yaml-p3 (Failed)
 2620 - f3_decaying_hit_8x8x8.4C.yaml-p2 (Failed)
 2621 - f3_decaying_hit_eb.4C.yaml-p2 (Failed)
 2622 - f3_decaying_hit_eb_revcond.4C.yaml-p2 (Failed)
 2624 - scatra_forced_hit_iso_8x8x8_dsm.4C.yaml-p3 (Failed)
 2625 - scatra_forced_hit_iso_8x8x8_dsm.4C.yaml-p3-restart_from_same_input (Not Run)
 2626 - scatra_forced_hit_mean_8x8x8.4C.yaml-p3 (Failed)
 2627 - scatra_forced_hit_mean_8x8x8.4C.yaml-p3-restart_from_same_input (Not Run)
 2630 - sohex8_multiscale_macro.4C.yaml-nestedPar (Timeout)
 2631 - sohex8_multiscale_macro.4C.yaml-nestedPar-restart (Not Run)
 2632 - sohex8_multiscale_macro_2micro.4C.yaml-nestedPar (Timeout)
 2633 - sohex8_multiscale_macro_2micro.4C.yaml-nestedPar-restart (Not Run)
 2634 - tsi_heatconvection_monolithic.4C.yaml-nestedPar (Failed)
 2745 - sohex8_multiscale_macro-vtk.4C.yaml-p2 (Not Run)
 2771 - unittests_deal_ii.np3 (Failed)
 2772 - unittests_deal_ii (Failed)
 2773 - unittests_deal_ii.np2 (Failed)
 2782 - unittests_mat.np3 (Failed)

@maxfirmbach
Copy link
Contributor Author

@mayrmt Would you mind taking a look at the tests related to fsi and contact? Why they fail and if they are fixable in an easy way?
@isteinbrecher Would you mind taking a look at the beam related tests?
@m-frey Would you mind taking a look at the tests related to micro-macro and nestedPar?

@mayrmt
Copy link
Member

mayrmt commented Apr 3, 2025

@maxfirmbach I'll have a look. Not today, but hopefully tomorrow.

@mayrmt
Copy link
Member

mayrmt commented Apr 4, 2025

@maxfirmbach FSI tests seem to just violate the result tolerance a little bit. Given the changes in subdomains combined with additive Schwarz components in the preconditioner, I am inclined to accept these changes as ok and just adapt the test result or the tolerance accordingly.

Contact tests are a different story and need more investigation.

@mayrmt
Copy link
Member

mayrmt commented Apr 4, 2025

I took care of the FSI and FS3I tests. I have removed them from the list of failing tests above.

@mayrmt mayrmt changed the title Draft: [Trilinos] Move from Isorropia to Zoltan2 [Trilinos] Move from Isorropia to Zoltan2 Apr 8, 2025
@mayrmt
Copy link
Member

mayrmt commented Apr 10, 2025

Contact tests with rebalancing require a closer look, since they fail with a run time error in the graph partitioner.

@m-frey
Copy link
Contributor

m-frey commented Apr 11, 2025

Ill look into the micro-macro and the nestesPar

@maxfirmbach maxfirmbach force-pushed the 927-Move-from-isorropia-to-zoltan2 branch from 1a483fd to ce66a2b Compare April 22, 2025 09:35
@maxfirmbach
Copy link
Contributor Author

maxfirmbach commented Apr 25, 2025

@eulovi Seems like the implementation is correct from a conceptual point of view (even though the results of the partitioning differ). Would you mind working on the output related tests and fix them e.g. vtk? I think you are faster with this than I am ... 😉

@eulovi
Copy link
Contributor

eulovi commented Apr 29, 2025

I fixed the following vtk output tests:

  • beam3eb_static_beam_to_solid_volume_meshtying_beam_to_beam_contact_boundingvolume-vtk-beam-interaction-geometric-search.dat-p2
  • embeddedmesh_parallel_bending_beam_fine-vtk-structure.dat-p3
  • beam3r_herm2line3_static_beam_to_solid_surface_contact_ironing_penalty_gap_variation_segmentation_lin_quad-vtk-averaged-normals.dat-p3
  • beam3r_herm2line3_static_beam_to_solid_surface_contact_mortar_line2_gap_variation_reference_segmentation_penalty_lin_quad-vtk-averaged-normals.dat-p3
  • elch_3D_tet4_s2i_butlervolmer_plating_monolithic_reg_none_vtk.dat-p3
  • membrane_patch_new_struct_vtk.dat-p2
  • poromultielastscatra_2D_quad4_tumor-vtk-porofluid.dat-p3
  • solid_runtime_ghosting_output-vtk-solid.dat-p3
  • solid_runtime_nurbs-vtk-solid.dat-p3
  • solid_runtime_velocity_output-vtk-solid.dat-p2
  • solid_runtime_vtk_2d-vtk.dat-p2
  • solid_runtime_vtk_offset-vtk.dat-p2

For the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 I wasn't sure if I should change it or not as the results of the beams are
different:
beam-results

If this is expected with the current changes, let me know to add the current VTK results.

@maxfirmbach
Copy link
Contributor Author

@eulovi Thanks for fixing some of the vtk related tests! Did you already remove the tests from the list above? Would you mind doing that?

@maxfirmbach
Copy link
Contributor Author

@isteinbrecher Any opinion on:

For the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 I wasn't sure if I should change it or not as the results of the beams are
different

@isteinbrecher
Copy link
Contributor

I think the changes in the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 are too large. Do the result test of the "main" simulation pass?

@maxfirmbach
Copy link
Contributor Author

maxfirmbach commented Apr 30, 2025

@isteinbrecher @eulovi The results in the "main" simulation pass, but there's a Epetra ERROR -1, packages/epetra/src/Epetra_CrsMatrix.cpp feature appearing ... so not sure what influence this has (also happens on the current main branch). Every time we skip these errors something wrong is happening, so I assume that also here we do something wrong e.g. accessing a index not appearing in a map or not locally available ... etc.

@isteinbrecher
Copy link
Contributor

Is it possible to identify where this error occurs?

@maxfirmbach
Copy link
Contributor Author

@davidrudlstorfer Yeah, I messed up something ...

@davidrudlstorfer
Copy link
Contributor

I was able to restore the branch/commit via git reflog (thanks @amgebauer!)

@maxfirmbach see https://github.com/davidrudlstorfer/4C/tree/927-Move-from-isorropia-to-zoltan2

@maxfirmbach
Copy link
Contributor Author

@davidrudlstorfer @amgebauer Cool! Sorry again ...

@maxfirmbach maxfirmbach force-pushed the 927-Move-from-isorropia-to-zoltan2 branch from 2533c5d to 66ca65f Compare October 9, 2025 06:47
maxfirmbach and others added 23 commits October 14, 2025 15:09
Signed-off-by: Max Firmbach <max.firmbach@unibw.de>
Tolerances can be loosened, since the solver tolerances are actually a
couple of orders of magnitudes above the result tolerances.
Tolerances can be loosened, since the solver tolerances are actually a
couple of orders of magnitudes above the result tolerances.
These tests target the contact preconditioner, thus their nonlinear
solver tolerance is way to loose. Hence, results are affected by changes
in the subdomain. Slightly loosening the result tolerances makes the
test more robust.

Number of linear iterations have changed as well.
Change in partioning has reduced the number of linear iterations.
@maxfirmbach maxfirmbach force-pushed the 927-Move-from-isorropia-to-zoltan2 branch from 289b5b0 to 7f4f97e Compare October 14, 2025 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: in progress Issues that are currently being worked on taskforce: tpetra Issues related to the migration from Epetra to Tpetra team: discretization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants