ENSIME support

[fommil]

Quite a few people have asked if they could use ENSIME on projects that are built with Mill, so I'm interested in finding out what needs to be done to make that happen.

Ideally the user could install a plugin that is automatically included by every Mill build, which adds

-Xplugin:${sys.props("user.home")}/.cache/ensime/lib/ensime-${scalaVersion}.jar

to the scalac options for every file that is built in the project.

However, I don't believe Mill has the concept of a globally installed user plugin, like sbt has, so it might be worth exploring adding support for that to Mill itself. The support would be inert by default. e.g.

if "~/.cache/ensime" exists then {
  // can assume that the user wants to have ENSIME enabled
  if "~/.cache/ensime/lib/ensime-$scalaversion.jar" exists then {
    scalacOptions += ...
  } else {
    print("message about how to install specific version of ensime, or suppress this message")
  }
}

do you have any ideas on how to do it without requiring changes to Mill itself?

[lefou]

If all you need to support ENSIME from Mill is the use of a scalac compiler plugin, that it should be easy to add. A solution that can work without any changes to Mill itself could be a custom trait which just adds the required options. That would mean, an ENSIME user need to mix-in this trait into her ScalaModules.

We had some comparable support for Metals to provide semanticDB data, before we provided built-in support for semanticDB. Have a look at the now deprecated trait mill.scalalib.bsp.ScalaMetalsSupport.

A trivial implementation could look as follows:

trait EnsimeSupport extends ScalaModule {
  override def scalacPluginIvyDeps: Target[Agg[Dep]] = T {
    super.scalacPluginIvyDeps() ++ Agg(
      ivy"..." // the Maven or Ivy GAV of your plugin
    )
}

Of course, you can make that much more sophisticated.

If your plugin is not consumable via Maven/Ivy, you need to add it to scalacPluginClasspath manually and also tweak the scalacOptions. But I do not advice to go that route, as Scalac compiler plugins need to be built against the full compiler version release, and consuming the right plugin is much easier as managed dependency.

You would use it as follows:

object foo extends ScalaModule with EnsimeSupport {
  // ...
}

I also recommend to read the various discussions that led to the current built-in design, esp. to understand why we don't support some side-injections of scalac options.

• Better Metals support - Automatic SemanticDB enablement (RFC) #1546
• SemanticDB support to be used by BSP (and others) #1977

[lefou]

Yeah, it was just a comment and I clearly did not wanted to implement it for you, haha. I just wanted to make clear, what's in my eyes possible as a thirdparty plugin.

As your compiler plugin produces extra files, it clearly modifies the returned result (in comparison to a compilation without that plugin enabled), which is a mill.scalalib.api.CompilationResult. (If you are curious, it contains a mill.api.PathRef of the created classes directory, which is sensitve to the contents of all files in it.)

Given an appropriate implementation and following my above outlined proposal, instead of compiling a fictive project with

$ mill showNamed __.compile
{
 "foo.compile": {
    "analysisFile": "/tmp/example/out/foo/compile.dest/zinc",
    "classes": "ref:4a69aea9:/tmp/example/out/foo/compile.dest/classes"
  },
  "bar.compile": {
    "analysisFile": "/tmp/example/out/bar/compile.dest/zinc",
    "classes": "ref:c984eca8:/tmp/example/out/bar/compile.dest/classes"
  }
}

it could instead be compiled by running the new proposed command

$ mill --import <???> showNamed Ensime/compileWithEnsimePlugin __.compile
{
  "Ensime.compileWithEnsimePlugin": [
    {
      "analysisFile": "/tmp/example/out/Ensime/compileWithEnsimePlugin.dest/foo/zinc",
      "classes": "ref:86ade12b:/tmp/example/out/Ensime/compileWithEnsimePlugin.dest/foo/classes"
    },
    {
      "analysisFile": "/tmp/example/out/Ensime/compileWithEnsimePlugin.dest/bar/zinc",
      "classes": "ref:e312a4f0:/tmp/example/out/Ensime/compileWithEnsimePlugin.dest/bar/classes"
    }
  ]
}

The output is completely made up though.

[fommil]

As your compiler plugin produces extra files, it clearly modifies the returned result (in comparison to a compilation without that plugin enabled), which is a mill.scalalib.api.CompilationResult

I don't believe that is the case. Mill doesn't, and shouldn't, know anything about what the compiler plugin is doing. Just treat it like any other parameter passed to the compiler. Having a separate compilation invocation for the ensime plugin would defeat the design which reduces the power usage of the dev cycle.

I'd very much like to implement it as a third party source plugin, but it doesn't seem to me like this is possible. In sbt I can put the plugin into ~/.sbt/1.0/plugins/ making it a user-installed plugin, not a project-installed plugin. Mill doesn't seem to have an equivalent. I suppose I could investigate monkey patching the Mill classpath to inject some arbitrary code.

[lefou]

As your compiler plugin produces extra files, it clearly modifies the returned result (in comparison to a compilation without that plugin enabled), which is a mill.scalalib.api.CompilationResult

I don't believe that is the case. Mill doesn't, and shouldn't, know anything about what the compiler plugin is doing. Just treat it like any other parameter passed to the compiler.

You told me, that your plugin is creating extra files and I was assuming, those files will be placed inside the classes directory. So, the addition of the compiler plugin will effectively alter the return value of the compile target, won't it?

I'd very much like to implement it as a third party source plugin, but it doesn't seem to me like this is possible. In sbt I can put the plugin into ~/.sbt/1.0/plugins/ making it a user-installed plugin, not a project-installed plugin. Mill doesn't seem to have an equivalent. I suppose I could investigate monkey patching the Mill classpath to inject some arbitrary code.

Mill is not sbt, we typically don't accept untrackable inputs. We aim for reproducible builds and like to see the build.sc as single source of truth.

I already told you one way to get thirdparty plugins into your build. Another way is to use a default predef.

[fommil]

your plugin is creating extra files and I was assuming, those files will be placed inside the classes directory.

nope, they go into ~/.cache/ensime and are kept entirely away from the build. Apologies, I should have made that clear. From the build tool's perspective, they don't exist and shouldn't be tracked.

[lefou]

I suggested two ways:

• create an non-invasive external module to generate ~/.cache/ensime data, which might be a bit less performing effectively
• create a Ensime trait the user can mix-in to unleash the full performance, but requires editing the build.sc

Both ideas may already help those people who demanded Mill support. IMHO, it's worth to explore these spaces before exploiting Mill internals further.

I'm not strictly against a deeper integration, but I'm rather curious to not loose any of the other qualities of Mill. I guess, the best way would be to gather feedback and experience from the proposed solutions. And after that start digging deeper (if required) with a RFC-like design or a pull request, so we can discuss details and explain reservations.

[lefou]

Can you describe, how your compiler plugin and your editing experience works? How are dependencies handled? How do you support cross-setups where alternative source files for the same final class exists?

[fommil]

that's all covered in detail in the README of the .tar.gz at https://ensime.github.io/ if you're interested in this.

[fommil]

Project-level plugin. It would be great to see this added to the core Mill distribution so that ENSIME users don't have to do anything.

The check at

    val ensimeWanted = os.exists(os.home / ".cache" / "ensime")

can be safely, and quickly, used to enable/disable all the logic.

import mill._
import mill.scalalib._

// To install this Mill plugin, add
//
// ```
// import $file.plugins.Ensime
// import Ensime.EnsimeModule
// 
// object ... extends EnsimeModule
// ```
//
// to your build.sc

trait EnsimeModule extends ScalaModule { module =>

  def ensimeJar : T[Option[PathRef]] = T.input {
    val jar = os.home / ".cache" / "ensime" / "lib" / s"""ensime-${scalaVersion()}.jar"""
    val ensimeWanted = os.exists(os.home / ".cache" / "ensime")
    if (os.isFile(jar) && ensimeWanted) {
      // make sure the user always has sources if ENSIME is enabled
      val fetchTask = fetchDepSources()
      fetchTask()
      Some(PathRef(jar))
    } else {
      if (ensimeWanted){
        T.ctx.log.error(s"ENSIME not found. Try\n\n      sbt ++${scalaVersion()}! install\n\nin the ensime-tng repo.")
      }
      None
    }
  }

  override def scalacOptions = T {
    super.scalacOptions() ++ {ensimeJar() match {
      case Some(jar) => Seq(s"-Xplugin:${jar.path.toIO.getAbsolutePath}")
      case None => Seq()
    }}
  }

  private def fetchDepSources: mill.define.Task[() => Unit] = T.task {
    import coursier._
    import coursier.util._
    import scala.concurrent.ExecutionContext.Implicits.global

    val repos = module.repositoriesTask()
    val allIvyDeps = module.transitiveIvyDeps() ++ module.transitiveCompileIvyDeps()
    val coursierDeps = allIvyDeps.map(module.resolveCoursierDependency()).toList
    val withSources = Resolution(
      coursierDeps
        .map(d =>
          d.withAttributes(
            d.attributes.withClassifier(coursier.Classifier("sources"))
          )
        )
        .toSeq
    )
    () => {
      val fetch = ResolutionProcess.fetch(repos, coursier.cache.Cache.default.fetch)
      val _ = withSources.process.run(fetch).unsafeRun()
    }
  }
}

[lefou]

You can add a new module to the contrib section and open a PR. This will result in some user convenience but will couple your plugin tightly to Mill releases. It does not automatically mean, Mill maintainers will maintain it, but if it has a test suite, it would help to keep it functional.

Alternatively, you can develop and publish it external to Mill. This means, you can support older Mill versions and also newer versions of ENSIME without waiting for a new Mill release.

[fommil]

Thanks for the offer. However, if the user still needs to extends Ensime if it's distributed under contrib, then I'm not sure it makes any sense to have it distributed as part of Mill.