Bazel green build CI job now fails persistently due to rules_java versioning

[mbland]

All CI jobs except "Bazel green head" pass for my PRs working towards #1482 with the following error:

bazel --nosystem_rc --nohome_rc info
ERROR: Failed to load Starlark extension '@@rules_java//toolchains:toolchain_utils.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @@rules_java
This could either mean you have to add the '@@rules_java' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
ERROR: cannot load build configuration key because of this cycle
Traceback (most recent call last):
  File "bazelci.py", line 4588, in <module>
    sys.exit(main())
  File "bazelci.py", line 4556, in main
    execute_commands(
  File "bazelci.py", line 1321, in execute_commands
    bazel_version = print_bazel_version_info(bazel_binary, platform)
  File "bazelci.py", line 1623, in print_bazel_version_info
    execute_command(
  File "bazelci.py", line 2750, in execute_command
    return subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bazel', '--nosystem_rc', '--nohome_rc', 'info']' returned non-zero exit status 2.

After reviewing my notes from cd22d88, under Bump to rules_java 7.9.0 for Bazel 7 compatibility, it seems like our WORKSPACE builds can't be compatible with Bazel 6/7 and Bazel 8 at the same time. The basic issue is that for rules_java > 7.9.0, the WORKSPACE prefix in Bazel 6/7 causes Java toolchains to register under a different type than what the rules are expecting later. This issue doesn't affect Bzlmod builds.

Would it be possible to disable the "Bazel green head" builds until the Bzlmod compatibility lands, and then reenable that build with Bzlmod enabled?

[simuons]

Hi, @mbland, I've changed last_green to latest in #1627
I understand it's not a robust solution, but I hope it will hold until bzlmod lands into rules_scala.
I'll ask you to rebase your PRs so I could merge them after review.

[mbland]

Sounds good, thanks! Did you want to keep this issue open until that point, or close it now, and/or open a new one to switch it back at that point?

[simuons]

Let's keep it open as actual issue is not fixed.

[mbland]

BTW, I pinpointed in a Bazel Slack thread with @shs96c the builds before and after the rules_java 7.9.0 bump from #1619:

• Passing Bazel green head rules_scala build
• Failing Bazel green head rules_scala build

And just to mention it explicitly here: bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix "fixes" the problem in Bazel 8, enabling WORKSPACE builds to use rules_java > 7.9.0. But as mentioned already, doing so would break WORKSPACE for Bazel 6 and 7, which still have rules_java_builtin in their WORKSPACE prefix.

[mbland]

I just discovered some further complications, noted in #1652, that may prevent building with the latest green Bazel build. , or require breaking WORKSPACE compatibility permanently.

Update: We could actually maintain WORKSPACE compatibility on Bazel 8 and rules_java 8, but it would no longer work with Bazel 6 or 7. If/when we decide to update for Bazel 8 compatibility, we could reenable this latest green head build, but probably not before.

Another update: I've created the bazel-8-compatibility working branch in my rules_scala fork with the commit message for mbland@ebe714d describing the current build breakages, which are the same under WORKSPACE and Bzlmod.

Note that this branch/commit is based on my bzlmod working branch, not the current upstream master branch.

[mbland]

I just filed #1671 to fix the latest Bazel builds in the wake of Bazel 8 being released an hour ago (which I noticed after pushing a commit to #1660) by setting the build to 7.x instead of latest.

On the bright side, per my work on both #1482 and #1652, I have a branch incorporating both Bzlmod and Bazel 8 compatibility changes that builds with Bazel 8.0.0, the latest Bazel 9 rolling release, and the latest Bazel green HEAD. So once that Bazel 8 release lands, we can restore the last_green build (or add it, keeping latest as well, as there might be utility to having both).