[SUREFIRE-1963] Detecting multiple test-frameworks

[jira-importer]

Slawomir Jaranowski opened SUREFIRE-1963 and commented

When multiple test frameworks are present on project classpath surefire chooses one framework for running test.

During automatically provider discover some of type test can be skipped without any knowledge and information for developer.

It is danger situation than developer can add second test framework to project and in the result some of test will be skipped silently.

We can force usage one or more needed providers by adding those to plugin dependency and we can have control on executing test.

Proposition

• add options multipleFrameworks empty (not set) by default, with possible value warn or fail

Scenario 1 - default

when

• multipleFrameworks = *empty* - not set
• many tests framework on classpath
• no providers on plugin dependency

then

• current behavior - choose first provider without warning about skipped test

Scenario 2

when

• multipleFrameworks = *warn*
• many tests framework on classpath
• no providers on plugin dependency

then

• print warn about multiple providers

Scenario 3

when

• multipleFrameworks = *fail*
• many tests framework on classpath
• no providers on plugin dependency

then

• print error about multiple providers and break build

Scenario 4

when

• many tests framework on classpath
• selected providers on plugin dependency

then

• ignore multipleFrameworks
• use providers from dependency list, no warning

---

Issue Links:

• SUREFIRE-2033 Regression: 3.0.0-M5 misidentifies JUnit 5 configuration as JUnit 4

• SUREFIRE-1458 When multiple surefire providers are available through the maven plugin dependency test are executed N times

• SUREFIRE-1527 Surefire does not discover junit5 test case when testng is in the classpath

• SUREFIRE-1911 Groovy-based JUnit 4 tests no longer run w/ 5 in classpath

• SUREFIRE-1527 Surefire does not discover junit5 test case when testng is in the classpath

• MINVOKER-285 Consider of using groovy-all once again

Remote Links:

• GitHub Pull Request #399
  

2 votes, 5 watchers

[jira-importer]

Robert James Oxspring commented

Is it really healthy that the default behaviour will be to continue to silently skip some tests? That way the problem of "silently skipping" tests by default is entirely unresolved by this issue.

Many people won't know of the multipleFrameworks option, and with this default behaviour won't even know if they're suffering from it. I can understand that failing their build might be considered harsh and surprising, but I'd have thought it should be a warning at a minimum.

[slawekjaranowski]

as we limit a providers implementation it will be not needed