Addition of functional/integration testing for CTSM

[ekluzek]

This is something that is useful both from a software testing angle as well as a science exploration, validation, verification, and tuning angle.

Software testing:

Software test levels contain:

1. Unit-testing to the subroutine or module level
2. Integration or functional level, several modules tested together
3. System testing

In terms of our testing we cover mostly "3" above the system level tests. But, this level can't cover everything because of it's expense and course level. We have a small amount of unit-testing for "1" -- but NO testing for the middle tier "2". So this is a hole in our testing strategy.

Here's the UCAR Software Engineering Assembly (SEA) document that goes over Integration testing briefly

https://sea.ucar.edu/best-practices/testing/integration

Better Scientific Software has this article on testing definitions

https://bssw.io/blog_posts/understanding-software-testing-practices

There's also a Wikipedia article on Integration Testing

Science Testing:

For scientific development

We can't really validate the whole system, but you can examine smaller parts of important processes that cover interlinked modules. This can give confidence that the smaller parts are put together correctly. In some cases you might be able to do things like check for particular cases that have analytic solutions as a verification technique. It can also be done to explore how a smaller part of the system works, as well as possibly tuning some parameters.

Article from Better Scientific Software:

https://bssw.io/blog_posts/better-science-through-software-testing

Examples, of scientists using integration testing

• Scientists that first implement in python to see behavior before implementing in FORTRAN
• Gordon Bonan's multi-level canopy offline driver
• The FAN PR (Fan 2019 ctsm #767) was designed to work with f2py so modules in the FORTRAN code could be called from python
• Martyn Clark's laugh tests https://journals.ametsoc.org/view/journals/hydr/22/6/JHM-D-20-0175.1.xml
• Allometry test in FATES: Allometry unit test NGEET/fates#1186

[adrifoster]

@ekluzek thanks for this! I think there is a pretty fuzzy line between unit and integration testing. But I'm not sure my allometry test is an integration test. It really is only testing each individual allometry equation/parameter set.

Because it's doing it in a non-pass/fail way, I suppose it's not really a true "unit test" - which I think are more geared towards logic, error-handling, etc.

However, I would not call it integration. I agree it's more functional, prototyping, or even similar to QA testing.