CTSM-FATES timing/cost

[dlawrenncar]

@rosiealice @ckoven @wwieder @ekluzek Following up on conversations we had about CTSM-FATES costs at the LMWG meeting. The new CSL allocation request is going to need to be written this summer and this would be good opportunity to try to establish at least general FATES costs, relative to big-leaf CTSM, for the purpose of writing the proposal. I'm aware that the costs will be more variable through time, but even a ballpark estimate would be helpful for perhaps the SP version and the full competition BGC version. I'd be happy to explore myself, which would give me opportunity to run FATES for the first time. Perhaps @rosiealice could point me to two relevant cases (SP, BGC full competition) to start with.

Definition of done:

• Upload timing results for FATES-SP
• Upload timing results for FATES-fixed-BGP

[ckoven]

@dlawrenncar if it helps, my historical transient runs took 20 minutes per decade at 4x5 resolution (so ~1000 gridcells, of which ~750 had living vegetation) on 288 cores on cheyenne.

output from that run is here: /glade/scratch/charlie/archive/fates_clm50_global_4x5_historicaltransient_2e3f469f_2905a9ba if you're interested.

[ckoven]

sorry, 20 minutes per year.

[dlawrenncar]

Thanks, I was going to try to establish costs for 1 and 2 deg simulations, presuming that 4x5 resolution is (a) cheap enough as to mainly be in the noise and (b) mainly for development/debugging purposes and not production.

…

On Tue, Jul 7, 2020 at 10:58 AM Charlie Koven ***@***.***> wrote: sorry, 20 minutes per year. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1076 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVAJ3FL7BJ6PDVO62VDR2NH4NANCNFSM4OTATHLA> .

[wwieder]

here's a recent 4x5 no crop with timing files for comparison /glade/scratch/wwieder/clm5_4x5_woodCN_cont_spin/run/timing On Tue, Jul 7, 2020 at 11:53 AM David Lawrence <notifications@github.com> wrote:

…

Thanks, I was going to try to establish costs for 1 and 2 deg simulations, presuming that 4x5 resolution is (a) cheap enough as to mainly be in the noise and (b) mainly for development/debugging purposes and not production. On Tue, Jul 7, 2020 at 10:58 AM Charlie Koven ***@***.***> wrote: > sorry, 20 minutes per year. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1076 (comment)>, or > unsubscribe > < https://github.com/notifications/unsubscribe-auth/AFABYVAJ3FL7BJ6PDVO62VDR2NH4NANCNFSM4OTATHLA > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1076 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5IWJDBPIMWQ4PRKV2IN2DR2NOK5ANCNFSM4OTATHLA> .

-- Will Wieder Project Scientist CGD, NCAR 303-497-1352

[ekluzek]

@dlawrenncar is going to work on this. Note, that you need to make sure that FATES is spunup well for your timing tests. FATES is more likely to have a change in timing cost as the model runs, unlike big-leaf CTSM.

[rosiealice]

Hi @dlawrenncar ,
I set these 4x5 simulations off before I saw your comment on the 1/2 deg simulations.

FATES Fixed biogeog with competition with 6 PFTs. 40y
/glade/scratch/rfisher/archive/fates_timin_6pft_fbg_comp/lnd/hist

CLM5 BGC no crop. 40y.
/glade/scratch/rfisher/archive/clm5_timing_bgc/lnd/hist

Broadly, the CLM simulation took 3 hours, and the FATES one took 18. It might be slightly longer given the spinup issues Eric mentioned. I can take a look at the latter years to check (lots to do today before they turn the computer off!!)

One interesting thing would be to look at how the FATES aggregation parameters affect the speed (because they should, especially for patch dynamics...

[dlawrenncar]

OK. Looking at the timing files, I see: FATES Fixed biogeog with competition with 6 PFTs /glade/work/rfisher/git/ctsmjuly20/cime/scripts/fates_timin_6pft_fbg_comp/timing 82 pe-hrs/yr (74 in first submission) 53 yrs/day CLM BGC no crop /glade/work/rfisher/git/ctsmjuly20/cime/scripts/clm5_timing_bgc/timing 14 pe-hrs/y 303 yrs/day So, as @rosiealice showed, the cost is ~6x. Obviously, that's a lot. I took a quick look through the timing files to see if I could see the source of the cost increase and I guess it will not be a surprise that my quick look suggests that the cost increase is mostly embedded within canflux. I guess this is simply due to the larger number of calls to photosynthesis due to the much larger number of patches compared to pfts. So, probably this is going to be hard to reduce without reducing the cost of the canopy flux calculations. At some stage, I will check again at 2deg resolution for the purpose of writing the next CSL allocation proposal. We should also have a conversation at some point soon about costs since this large increase in costs is going to significantly impact computational resource planning and will have implications for implementation in CESM3.

…

On Fri, Jul 10, 2020 at 2:31 AM Rosie Fisher ***@***.***> wrote: Hi @dlawrenncar <https://github.com/dlawrenncar> , I set these 4x5 simulations off before I saw your comment on the 1/2 deg simulations. FATES Fixed biogeog with competition with 6 PFTs. 40y /glade/scratch/rfisher/archive/fates_timin_6pft_fbg_comp/lnd/hist CLM5 BGC no crop. 40y. /glade/scratch/rfisher/archive/clm5_timing_bgc/lnd/hist Broadly, the CLM simulation took 3 hours, and the FATES one took 18. It might be slightly longer given the spinup issues Eric mentioned. I can take a look at the latter years to check (lots to do today before they turn the computer off!!) One interesting thing would be to look at how the FATES aggregation parameters affect the speed (because they should, especially for patch dynamics... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1076 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVF45RJZCSIWECSBKZLR23GVHANCNFSM4OTATHLA> .

[ckoven]

I'm actually a bit surprised that the number of patches should be greater than the number of PFTs. FATES currently limits it to 10 primary patches per site, which seems like less than 6x the number of PFTs in big-leaf CLM?

[rgknox]

For any given column, we seem to be operating with our 1600 cohorts each requiring their own photosynthesis calculations, a scheme which is embedded in the canflux iterative solve. So only 6X, yay.

[dlawrenncar]

Sorry, yeah, it is the much larger number of cohorts compared to CTSM PFTs that is the likely source, not the potentially larger number of patches. So, with much larger number of cohorts compared to PFTs, only 6x, yay! ... but still 6x, boo!

…

On Fri, Jul 10, 2020 at 9:55 AM Ryan Knox ***@***.***> wrote: For any given column, we seem to be operating with our 1600 cohorts each requiring their own photosynthesis calculations, a scheme which is embedded in the canflux iterative solve. So only 6X, yay. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1076 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVAH7YES5S2Q5AXGYSDR242V3ANCNFSM4OTATHLA> .

[rgknox]

For accuracy's sake, I just want to point out that we actually perform photosynthesis on each leaf-layer of each canopy-layer, and since we use PPA, each pft has its own leaf layer (sorry, I shouldn't had said cohort). So we end up doing photosynthesis on an array that is pft x leaf-layer x canopy layer. In order to make sure we hit all the possible layers, we DO loop by cohort, but we also do a lot of masking to avoid redundancy.

See here, we actually loop over leaf layers: https://github.com/NGEET/fates/blob/master/biogeophys/FatesPlantRespPhotosynthMod.F90#L375

That said, I think that reducing our number of leaf layers is something that could potentially reduce computation, and may be low hanging fruit.