CAM regression test is failing in lnd_set_decomp_and_domain.F90 when using ifx

[cacraigucar]

Brief summary of bug

The CAM CONUS test is failing in lnd_set_decomp_and_domain.F90 when using ifx

General bug information

CTSM tag: Using Bill's fork; hash c7230f0

Does this bug cause significantly incorrect results in the model's science? [Yes / No]
yes

Configurations affected: [Fill this in if known.]
CAM regression test: SMS_D_Ln9_P1536x1.ne0CONUSne30x8_ne0CONUSne30x8_mt12.FCHIST.derecho_intel.cam-outfrq9s

Details of bug

Getting floating invalid at line 510 of lnd_set_decomp_and_domain.F90

Important details of your setup / configuration so we can reproduce the bug

Using cam6_4_152 with Bill's CTSM and MOSART forks with fixes for ifx. [If you'd like to use my fork with these changes you can clone https://github.com/cacraigucar/CAM; hash 60bc7b0]

Important output or errors that show the problem

You can see my test at: /glade/derecho/scratch/cacraig/aux_cam_intel_20260219092032/SMS_D_Ln9_P1536x1.ne0CONUSne30x8_ne0CONUSne30x8_mt12.FCHIST.derecho_intel.cam-outfrq9s.GC.aux_cam_intel_20260219092032/run
NOTE: This used hash a61bea6 of my CAM fork, before a couple of CAM fixes were made in this branch.

[billsacks]

Interesting, thanks @cacraigucar . This one is failing differently from the other issues I've come across: it's dying inside esmf, in this call:

       ! create route handle to map ocean mask from mask mesh to land mesh
       call ESMF_FieldRegridStore(field_mask, field_lnd, routehandle=rhandle_mask2lnd, &
            srcMaskValues=(/srcMaskValue/), dstMaskValues=(/dstMaskValue/), &
            regridmethod=ESMF_REGRIDMETHOD_CONSERVE, normType=ESMF_NORMTYPE_DSTAREA, &
            srcTermProcessing=srcTermProcessing_Value, &
            ignoreDegenerate=.true., unmappedaction=ESMF_UNMAPPEDACTION_IGNORE, rc=rc)

and I think on this line deep inside some ESMF C code:

      *element += factor * *value;

Maybe there are some NaN values in the mesh that are being handled differently? We'll need to look into the various input files in this case.

[ekluzek]

I don't see NaN's in the CONUS mesh or the t232 mesh:

ncks --chk_nan /glade/campaign/cesm/cesmdata/inputdata/share/meshes/ne0CONUSne30x8_ESMFmesh_c20200727.nc
ncks --chk_nan /glade/campaign/cesm/cesmdata/inputdata/share/meshes/tx0.1v2_ESMFmesh_cd5_c20210105.nc

[ekluzek]

It does look like this is #3636 coming back to bite us. So it might be something like then where it sometimes works and sometimes fails because of some race condition. The ESMF update could also be at play here as well.

[ekluzek]

As part of #3769 I ran I compsets for this grid and it worked fine. So we have the same situation as we had in #3636 where it works standalone and fails coupled.

grep RUN *CONUS*/TestStatus
SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam6LndTuningMode_2013Start--clm-nofireemis.GC.ctsmscictsm5421ifxc_s_int/TestStatus:PASS SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam6LndTuningMode_2013Start--clm-nofireemis RUN time=161
SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam7LndTuningMode_2013Start--clm-nofireemis.GC.ctsmscictsm5421ifxc_s_int/TestStatus:PASS SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam7LndTuningMode_2013Start--clm-nofireemis RUN time=165
SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm60SpCrujra.derecho_intel.clm-clm60cam7LndTuningMode_2013Start--clm-nofireemis.GC.ctsmscictsm5421ifxc_s_int/TestStatus:PASS SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm60SpCrujra.derecho_intel.clm-clm60cam7LndTuningMode_2013Start--clm-nofireemis RUN time=163
SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm60Sp.derecho_intel.clm-clm60cam7LndTuningMode_2013Start--clm-nofireemis.GC.ctsmscictsm5421ifxc_s_int/TestStatus:PASS SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm60Sp.derecho_intel.clm-clm60cam7LndTuningMode_2013Start--clm-nofireemis RUN time=166

[billsacks]

I just got a failure in the case2 run of the prealpha test ERP_D_Ln9.ne30pg3_ne30pg3_mt232.F1850C_MTso.derecho_intel.cam-outfrq9s with the same traceback - line 510 of lnd_set_decomp_and_domain.F90, but with the actual death occurring deep inside of ESMF. This was from cesm3_0_alpha08e with an update in CMEPS to a branch I'm working on, but no machine changes - so using ifort, not ifx. A rerun failed in the same way.

See run directory here: /glade/derecho/scratch/sacks/cesm3_0_alpha08e_handlerank3_prealpha/ERP_D_Ln9.ne30pg3_ne30pg3_mt232.F1850C_MTso.derecho_intel.cam-outfrq9s.C.20260223_104249_79z9md/run/case2run

Given that the initial run passed both times, I wonder if this is a memory issue. (In this test, the initial run uses 512 tasks, 1 thread per task; case2 uses 256 tasks, 1 thread per task.)

ERP_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_MTso.derecho_intel.cam-outfrq9s_cam7_presc_volc also failed in the same way in run2 - see /glade/derecho/scratch/sacks/cesm3_0_alpha08e_handlerank3_prealpha/ERP_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_MTso.derecho_intel.cam-outfrq9s_cam7_presc_volc.C.20260223_104249_79z9md/run/case2run.

@fischer-ncar - also bringing you into the loop here.

[ekluzek]

I just got a failure in the case2 run of the prealpha test
.
.
.
Given that the initial run passed both times, I wonder if this is a memory issue. (In this test, the initial run uses 512 tasks, 1 thread per task; case2 uses 256 tasks, 1 thread per task.)

That could be the problem all along. We are using more nodes for standalone cases than CAM is using for the coupled cases. 13 nodes for ne30 and 12 for CONUS. Part of why we use so many nodes for standalone is just to have one node of the DATM to NOT be running that much faster than CLM does.

I think maybe I should have a standalone PE layout for some of these CAM cases that look closer to what CAM is actually running. @cacraigucar could CAM use more nodes for these cases than you are running now?

[billsacks]

I reran the two failing tests (ERP_D_Ln9.ne30pg3_ne30pg3_mt232.F1850C_MTso.derecho_intel.cam-outfrq9s and ERP_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_MTso.derecho_intel.cam-outfrq9s_cam7_presc_volc and they passed upon rerun (even though the first one failed again in the first rerun). I had started to get nervous that they were failing due to some change I made in CMEPS, since they both passed on the first try when I backed out cmeps to the version in cesm3_0_alpha08e, but this behavior of sometimes-failing-and-sometimes-passing makes me think this is just a sporadic issue and I don't think I'm going to go down the rabbit hole of trying to figure out whether and how my recent CMEPS changes might be related. That said, if others start to notice this happening more after updating to a future CMEPS tag (for a PR I'm about to open but haven't yet), then we can revisit it.