replace crate_universe with a gazelle plugin?

[ashi009]

We have a monorepo that has hundreds of crates in it. Using current crate_universe rules is painful in different ways:

• crates_repository
  • frequent and painfully slow splicing/repinning given that cargo touches Cargo.lock whenever someone adds a in-repo dep in their crate.
  • never a clean git merge/revert given the checksum in cargo-bazel-lock.json is not patchable
• crates_vendor in remote mode
  • no more waiting on splicing/repining
  • clean git merge/revert at most of the time
  • unreadable defs.bzl in the generated vendor_path dir
  • slow calls to all_crate_deps in BUILD files
  • chicken and egg issue after creating a new crate
    1. one must export Cargo.toml in the new crate for crates_vendor to use in manifests attr (separate issue)
    2. NOTE: one must not add all_crate_deps() as the deps are not in the generated all_crates_deps yet, and build file eval will fail when running crates_vendor
    3. run crates_vendor to regenerate all_crates_deps
    4. update rust_library to use all_crates_deps again

This is way behind the UX from rules_go with gazelle.

Can we add a plugin to read cargo metadata output and generate all the rules directly -- for both 3rd party ones and in-repo ones. So that we can sunset most parts of the crate_universe and provide a unified bazel experience to users.

https://github.com/Calsign/gazelle_rust has a gazelle plugin for rust but seems a little bit over-engineered.

[DavidZbarsky-at]

If you don't need to build with cargo, I suggest to have a single Cargo.toml file at the repo root instead of one per rust library. That way you will not invalidate it as you add/remove deps from libraries, as long as it is an in-repo dep. Our experience got much better once we did that. In addition, if you move to MODULE.bazel instead of WORKSPACE, you won't need a cargo-bazel-lock.json file anymore, the extension manages it internally to the bazel output dir, so you won't have git conflicts there.

[marvin-hansen]

Vendor remote is somewhat half-finished. I had quite some issues with it and eventually moved to vendor local as it is way more consistent, robust, reliable, and a lot faster. The different vendor modes have been discussed here:

bazelbuild/examples#492 (comment)

From my own experience building over 75 crates in a mono-repo, cross compiled to 2 targets, I rarely ever see more than one second delay before actual compilation starts. To get there, I configured the following:

• from_spec dependencies
• local vendoring
• All bazelmod
• I still maintain a Cargo.toml to preserve tooling currently unavailable in rules_rust, so I get the best of both worlds.

The exact configuration is replicated in this demo repo

By the time you reach 50 or more crates, from_cargo becomes painfully slow and there isn't terrible much you can do about due to the way it is implemented meaning you really want to switch to use from_spec and local vendoring.

I don't think this is an issue with the rules itself since you can use any of the four different vendor modes. I would say this is more of a documentation issue since not everyone knows that rules rust comes with four different vendor modes and each one has its own set of tradeoffs.