Testthat reporting success when it should report failure.

[mattfidler]

I see the following in the test that log:

Start test: linear oral model gives extraCmt=2
  test-lincmt-solve.R:512:7 [success]
End test: linear oral model gives extraCmt=2

Model:

V=THETA[1];
CLx=THETA[2];
Ka=THETA[3];
K=CLx/V;
C2=linCmtA(rx__PTR__,t,0,1,2,K,V,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,Ka,0.0,1.0,0.0,0.0);

Error: lincmt-solve.R:1:1 [error]
Start test: loading and unloading works

But testthat reports this as a success:

https://github.com/nlmixrdevelopment/RxODE/actions/runs/669412373

While I don't want the tests to stop on an error I want the whole test suite to fail if there is one error hence I use stop_on_failure = FALSE.

In testthat 2 this did error, but now it does not. Is this new? It isn't entirely clear in the documentation what should happen with this option when the entire test suite is run.

Still failures seem inconsistent. Sometimes errors still cause testthat to fail for the project, sometimes they pass with errors. Is this error ignored since it is outside a testthat block?

Thank you for the package and your time.

[hadley]

Could you please create a reprex as a minimal package that demonstrates the problem?

[mattfidler]

I'm afraid I don't know enough about the root of the problem to create a minimal package that demonstrates the problem.

If you think it is too much of an edge case to support please let me know

[hadley]

You don't need to know anything about the problem in order to create a minimal reprex — you just have to either start with a simple package and add the stuff you think contributes to it, or take the existing package and delete until it goes away.

[mattfidler]

I will see if I can get some time to do this. Right now I'm a bit resource constrained.

[hadley]

No rush, I'm unlikely to look at it for weeks to months anyway 😄

[hadley]

Closing since it's now 2 years later 😄