com.spun.util.FormattedException: Didn't find ... under target/fork_dir_2 when running from maven-surefire-plugin

[d-led]

Problem

when running approval tests (JUnit5, Java 17) from mvn test via the maven-surefire-plugin, the tests fail with

com.spun.util.FormattedException: Didn't find com....BlaBlaTest under .../target/fork_dir_2
        at com.spun.util.ClassUtils.getSourceDirectory(ClassUtils.java:103)
        at com.spun.util.ClassUtils.getSourceDirectory(ClassUtils.java:108)
        at com.spun.util.tests.TestUtils.getInfo(TestUtils.java:195)
        at com.spun.util.tests.TestUtils.getCurrentFileForMethod(TestUtils.java:182)
        at com.spun.util.tests.TestUtils.getCurrentFileForMethod(TestUtils.java:174)
        at org.approvaltests.namer.StackTraceNamer.<init>(StackTraceNamer.java:17)
        at org.approvaltests.Approvals$1.load(Approvals.java:41)
        at org.approvaltests.Approvals$1.load(Approvals.java:38)
        at org.approvaltests.Approvals.createApprovalNamer(Approvals.java:265)
        at com.spun.util.ArrayUtils.getOrElse(ArrayUtils.java:309)
        at org.approvaltests.core.Options$FileOptions.getNamer(Options.java:120)
        at org.approvaltests.Approvals.verify(Approvals.java:191)
        at org.approvaltests.Approvals.verify(Approvals.java:50)
        at org.approvaltests.Approvals.verify(Approvals.java:46)
        at com....BlaBlaTest.unwrapping_for_storage_format(BlaBlaTest.java:34)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[ERROR] com....BlaBlaTest.some_test

with maven, approvals and the plugin being latest ones. Earlier surefire plugin versions did not produce an insight into this stacktrace.

In IntelliJ the test run succeeds without any changes. Surefire seems to have a test run isolation feature, enabling the forks which might need a special treatment by approvals.

A work-around which helped me proceed: ↓

Workaround

<plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${surefire.version}</version>
        <configuration>
            <workingDirectory>.</workingDirectory>
            <forkCount>0</forkCount>
            <!-- both config values make sure the tests pass in a controlled fashion -->
        </configuration>
</plugin>

[ScottBob]

Not entirely sure how to replicate this as it is currently working on all of our GitHub actions. When we add your workaround it breaks things, so we don't want to do that. We'd be happy to join a call with you to understand the issue better.

[d-led]

I'll try to replicate in a clean repo. Will reopen if I can reproduce in an isolated fashion.

[d-led]

Here's a reproduction https://github.com/d-led/reproduce-approvals-bug/actions/runs/12947688141/job/36114785421#step:4:1644

the culprit is the surefire-recommended setting for test run isolation:

<workingDirectory>target/fork_dir_${surefire.forkNumber}</workingDirectory>

in case parallel execution does get triggered, e.g. via parallelizing test methods

<parallel>methods</parallel>

↓

Error:    StringRepeaterTest.repeating_once:13 » Formatted Didn't find org.example.examples.StringRepeaterTest under /home/runner/work/reproduce-approvals-bug/reproduce-approvals-bug/target/fork_dir_1
Error:    StringRepeaterTest.repeating_twice:18 » Formatted Didn't find org.example.examples.StringRepeaterTest under /home/runner/work/reproduce-approvals-bug/reproduce-approvals-bug/target/fork_dir_1

The setting was set by a parent POM, discovered via some-mvn-wrapper help:effective-pom > effective.xml

[ScottBob]

Fixed in version 24.16.0

Let us know if there is anything else.

[d-led]

@ScottBob thanks, will check. I guess a dependency needs to be upgraded due to a known CVE: https://mvnrepository.com/artifact/com.approvaltests/approvaltests/24.16.0

[d-led]

Fixed in version 24.16.0

Let us know if there is anything else.

it seems some other fix is needed. The build is still broken: https://github.com/d-led/reproduce-approvals-bug/actions/runs/13203995838/job/36862663432#step:4:1593

[ScottBob]

Please take a look at d-led/reproduce-approvals-bug#1. Does this address your concern?

[d-led]

Please take a look at d-led/reproduce-approvals-bug#1. Does this address your concern?

It looks as if it did solve the problem. Will look at it later. Something for the docs?