Add integration tests to check each supported scalac option against each supported scalac version

[satorg]

This is a call-out to trigger a discussion rather than an issue itself.

We could create an IntegrationTest configuration and add tests that would verify whether a certain scalac option is really supported for a specific scalac version.

Checking it by hand is quite boring and unreliable and it would be nice to automate it somehow.
A straightforward solution that comes to mind is using scala-cli for each option and every scala version the project supposed to support:

$ scala-cli compile -S 2.12.12 -O -Xlint:deprecation -- empty.scala
Compiling project (Scala 2.12.12, JVM)
Error: 'deprecation' is not a valid choice for '-Xlint'
Compiled project (Scala 2.12.12, JVM)

whereas

$ scala-cli compile -S 2.12.13 -O -Xlint:deprecation -- empty.scala

compiles without errors.

So would it be possible to employ scala-cli for integration tests in the project and does it look reasonable overall?

See also #5 for a related discussion.

[armanbilge]

👍 to the idea. But I propose we just do it with scalac; scala-cli is an extra dependency to juggle with its own complications (like that it requires JDK 17 ....)

[TonioGela]

👍 to the idea. But I propose we just do it with scalac; scala-cli is an extra dependency to juggle with its own complications (like that it requires JDK 17 ....)

Random thoughts: with coursier you can install a specific scalac version with cs install scalafmt:2.4.2. I bet itr can be used as a dependency very much we do in the tests of the typelevel toolkit, and with that we could theoretically download and install specific scalac version to use, with the proper flags, against a specific scala file.

Another chance would be leveraging sbt scripted tests maybe?

[armanbilge]

scalac itself is a very ordinary dependency :) No need to fuss with coursier. We can just add every version of scalac to the build and then invoke them in isolated classloaders or something.

[TonioGela]

scalac itself is a very ordinary dependency :) No need to fuss with coursier. We can just add every version of scalac to the build and then invoke them in isolated classloaders or something.

Okay, I'll sort out something then :)