Cycling over multiple years of forcing

[mnlevy1981]

@ElizabethYankovsky and I are setting up an ocean-only single-column case based on MOM6-example's single_column/KPP that uses MARBL to provide BGC tracers (which currently exists on a branch of my fork of MOM6). We have daily output from a CESM-MOM6 run that we are using to generate the forcing fields, and use data_override to read them into the model:

$ cat data_table
# gridname, fieldname_code, fieldname_file, file_name,       ongrid, factor
"OCN" , "lw",      "LWDN",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "sw",      "SWDN",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "rain",    "RAIN",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "evap",    "EVAP",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "sens",    "SENS_HEAT_FLUX", "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "taux",    "TAUX",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "tauy",    "TAUY",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "snow",    "SNOW",           "./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc", .true., 1.0
"OCN" , "runoff",  "", "", .true., 0.0
"OCN" , "calving", "", "", .true., 0.0

If we provide a single year of daily forcing output, the model runs successfully:

$ ncdump -h MARBL_FORCING_FILE_COLUMN.nc |
grep time
	time = UNLIMITED ; // (365 currently)
	double time(time) ;
		time:_FillValue = 10000000000. ;
		time:units = "days since 1982-01-01 00:00:00" ;
		time:time_origin = "01-JAN-1982 00:00:00" ;
		time:calendar = "NOLEAP" ;
		time:cartesian_axis = "T" ;
		time:modulo = " " ;

If we try to extend the forcing to cycle over 5 years of forcing (the length of time has increased from 365 to 1825 and the modulo attribute of time has changed from " " to 1825.), then our forcing file looks like:

$ ncdump -h MARBL_FORCING_FILE_COLUMN.nc |
grep time
	time = UNLIMITED ; // (1825 currently)
	double time(time) ;
		time:_FillValue = 10000000000. ;
		time:units = "days since 1982-01-01 00:00:00" ;
		time:time_origin = "01-JAN-1982 00:00:00" ;
		time:calendar = "NOLEAP" ;
		time:cartesian_axis = "T" ;
		time:modulo = 1825. ;

And the model crashes very early in the run (I would presume this is during the first time step):

FATAL: time_interp_external 2: period of list exceeds modulo period,file=./INPUT/CESM_MOM_COLUMNS/BATS/MARBL_FORCING_FILE_COLUMN.nc,field=TAUX

Image              PC                Routine            Line        Source
MOM6               0000000001C421E6  Unknown               Unknown  Unknown
MOM6               00000000013E9A1C  mpp_mod_mp_mpp_er          71  mpp_util_mpi.inc
MOM6               00000000011BD297  time_interp_exter         889  time_interp_external2.F90
MOM6               00000000010E897A  data_override_mod         974  data_override.F90
MOM6               00000000010EDFAA  data_override_mod         580  data_override.F90
MOM6               000000000103064B  mom_data_override          59  MOM_data_override_infra.F90
MOM6               0000000000A1572E  mom_surface_forci         853  MOM_surface_forcing.F90
MOM6               00000000009FCBF4  mom_surface_forci         280  MOM_surface_forcing.F90
MOM6               000000000087937E  MAIN__                    455  MOM_driver.F90

What does period of list exceeds modulo period mean? In the 365-day forcing, time = [0.5, 1.5, ..., 363.5, 364.5] and in the 5-year forcing, time = [0.5, 1.5, ..., 1823.5, 1824.5]. Am I missing something in MOM_input to specify the cycle for the forcing is more than a year?

Note that I am building this out of a CESM+MOM sandbox, which is based on FMS v2021.03.01 and an NCAR-specific MOM6 tag. I've done some digging, and am beginning to suspect that if the time dimension has a modulo attribute then it is treated to mean "this is an annual cycle" (regardless of the value of the attribute) but I would love to be wrong about that.

[raphaeldussin]

this is a FMS-related issue. Nothing in the FMS code suggests looping over multiple years is forbidden. There is something about looping over leap years here https://github.com/NOAA-GFDL/FMS/blob/ba9c712825372eb3dc703685dd5c7e9c7560faa3/time_interp/time_interp.F90#L131 maybe worth trying even though your calendar is set to NOLEAP

[mnlevy1981]

Thanks for this suggestion -- this led me to dig in to the calling tree a little bit more and it looks like the real issue is that the time_interp() interface we are using is calling time_interp_list() when we actually want to call time_interp_modulo(). I haven't gotten all the way through the code, but I suspect there is something about the format of our forcing file itself that determines which branch of the code we traverse; hopefully we just need to make a small change to some of the attributes in our netCDF file.

Once we determine what those small changes are, I'll report back so others in this position can avoid this particular pitfall.

[anthony-meza]

did you find any solutions to this? I'm interested in running some repeat year forcing experiments with MOM6