Run doctests in testing suite

[cgarling]

This should run the doctests as part of the test suite. I don't expect this to pass now because of printing / environment stuff that we probably need to be more careful about when writing the doctests.

[giordano]

I don't understand why you need eval: https://github.com/EnzymeAD/Enzyme.jl/blob/c009fa593821cbd9017450a6721259f7dbc6ebf1/test/doctests.jl

[cgarling]

Thanks for the tip. I added the @eval approach to fix failures with Documenter.jl not recognizing the package module (see here). This problem only appeared when I switched to ParallelTestRunner so I may just be configuring something improperly. The errors are things like

│ ┌ Error: Failed to evaluate `CurrentModule = AstrochemicalYields` in `@meta` block.
│ │   exception =
│ │    UndefVarError: `AstrochemicalYields` not defined in `Main`
│ │    Suggestion: check for spelling errors or missing imports.
│ │    Hint: AstrochemicalYields is loaded but not imported in the active module Main.

I fixed this by explicitly importing the package into Main using @eval.

Looking at the code in Enzyme it's not clear to me why this error occurs in my project but not in Enzyme because Enzyme has a similar @meta block in their index.md file. Any tips would be appreciated

[giordano]

JuliaTesting/ParallelTestRunner.jl#68. It's not always necessary, but if the output depends on the exact module where the code is being evaluated, then something like that may be needed.

[icweaver]

Interesting, appreciate this info y'all

[icweaver]

Thanks for opening this, Chris! I think you may be hitting on a more general issue/convention/standard? that I've never really been quite clear on: the distinction between: "CI" tests, and "Documentation" tests.

We currently have these somewhat separated by design with our CI.yml and Documentation.yml in some of our repos, but not all. I assume this is an organizational decision to help separate out concerns, but I could be missing the larger picture that's going on. If we decide to continue with the separate workflows route, then would just adding a doc build badge serve the same purpose that this PR is trying to achieve?

This is something that I have been thinking about exploring more and potentially implementing ecosystem-wide, but just never took the time. Is this something that would be worth chatting about around the fireplace? Apologies if I am misunderstanding the overall intent here

[cgarling]

Without being overly philosophical, my main opinions are

1. Doctests are tests and so should be run with the rest of the test suite. I.e., if a source code change breaks a doctest, that should be considered a test failure and show up in the "CI" action, not the "Documentation" action
2. Since doctests are tests they should be counted in code coverage and should not be duplicated in other test suites (e.g. the main test directory)

For my own packages I usually run doctests in the main CI action (as I'm suggesting here) and then disable them on the "Documentation" action. Basically the "Documentation" action only builds the documentation and all the tests are run in the "CI" action. I find this to be a better separation of concerns (that way running tests and building documentation are truly separated).

If you want to start a fireplace thread you can but I'm also fine with this being a stylistic choice for individual maintainers to make with their repos.

[icweaver]

This is awesome, thanks Chris! These are exactly the kind of convos I've been wanting to have as an org, I'll go ahead and set up a fireplace thread then. While I'm in theory all for folks doing their own thing in their repos, I think there's still a lot of value (at least to new folk) in making these sorts of reasonings more transparent.

And who knows, maybe one day we'll develop our own org style as we become more and more cohesive.