Embed Scala source code parameters into .class files meta-data (for better handling of Scala libraries in IDEs)

[unkarjedy]

Original issue description in IntelliJ Scala Plugin YouTrack:
https://youtrack.jetbrains.com/issue/SCL-20896/Source-code-parameters-for-Scala-libraries

---

Short summary:
The issue: IDEs can’t reliably interpret Scala source files in libraries due to missing project-specific compiler configurations (like compiler version, compiler options, compiler plugins used during the compilation)
The proposal: Embed essential compiler configurations in metadata (e.g., via a @CompilerOptions annotation) to maintain accurate source interpretation without relying on build tool modifications.

---

Below is the copy of the original YouTrack ticket description:

Normally, Scala source code is interpreted in the context of a "project", so that each source file has an associated compiler version and compiler options, which might include -Xsource and -Xplugin.

This is true for both the Scala compiler and an IDE. However, things are different when it comes to external libraries - the compiler reads the bytecode, which is inherently unambiguous, whereas an IDE might also show the source code. In such a case, there's no way to determine the original project configuration and thus to unambiguously interpret the code.

It isn't possible to assume that all source code is Scala 3, because Scala 3 and Scala 2 must be interpreted differently. What's more, even Scala 2 now has the -Xsource:3 option, so there's no single "Scala 2" code. While it's possible to detect macro annotations in source code, compiler plugins are transparent, e.g. the kind-projector. Interestingly, the same is also true for distinguishing between different Scala 2.x versions, but this is less an issue because 2.x is compatible with 2.x+1, and the version of libraries must match the version of a project module - Scala 2.x versions are binary incompatible but source compatible, whereas Scala 2 & 3 / -Xsource:3 / compiler plugins are binary compatible but source incompatible.

A compiler doesn't need to compile the already compiled libraries again, so source JARs are not even downloaded. However, while an IDE can also read APIs from the bytecode, if a user navigates to a class or method, the IDE should be able to show the corresponding source code, which is thus downloaded. Although the IDE doesn't "compile" the source code in the usual sense, syntax highlighting requires lexical analysis, code folding / indent guides require parsing, navigation requires type inference, all of which depend on the "compilation" parameters.

Currently, we rely on a JAR name to determine whether it contains Scala 2 or Scala 3 sources. However, the heuristics is unreliable, and is not applicable to non-Courser/Maven JARs, or sources in a directory.

While we might detect the presence of .tasty files in a JAR, in principle, a single JAR might contain bytecode from different Scala versions (or modules compiled with different parameters). We might index the .class / .scala correspondence, but it's more complicated and is not applicable to -Xsource:3 or compiler plugins.

We may benefit from a more reliable solution.
For example, each JAR might include used compilation parameters in META-INF, either for the whole JAR, or for the particular source files, etc.
However, in this case the responsibility for bundling of the options would be transferred to the build tools.

Another option is to store (selected non-default) compiler options in a @CompilerOptions annotation, much like the existing @SourceFile annotation. The advantage is that this doesn't require modifying build tools, and the metadata is per-file.

The simplest solution would store the options just as strings.
There is also a potential room for discussion of minimizing the amount of data stored in the annotation (if it's considered an issue).
E.g. it could store only a subset of "essential" compiler options (then another task would be to identify what are the "essential") OR the compiler options could be encoded somehow.

[unkarjedy]

The same is actual for Scala 2, so I invite to @som-snytt, @lryt, @SethTisue to this discussion

[sjrd]

Also ping @adpi2, who I believe did something in this space for Metals.