Protobuf and gRPC Summary #26

aaliddell · 2021-11-30T23:21:13Z

aaliddell
Nov 30, 2021

Below is some follow up thoughts on the proto/grpc discussion from earlier. I hope I have made a fair representation of the various players, please correct me where I haven't.

What's the Problem?

Fragmentation: There is no single ruleset that solves proto/grpc for everyone in a satisfactory way today. Arguably there may never be due to the complexities below, but we can make users' lives as easy as possible and optimise for the happy-path. Where users are off the common path, it should be clear how the existing tools can be extended rather than reinventing the wheel.
Guidance: There is little guidance on when users should pick a specific set of rules and what the tradeoffs are for each. This is solvable through improvements in documentation.
Multiple Implementations: Some languages have no definitive implementation of proto or grpc (e.g Rust), which forces rulesets to either pick a winner or else support everything.

What Makes This Difficult?

Cross Cutting: Rulesets for proto/grpc need to know about both protoc and the language for which the plugins are generating files. This requires expertise in every language that a ruleset hopes to support, as well as expertise in how protoc itself works, which narrows the field of people interested in working on this.
Weird Plugins: Protoc plugins are effectively a wild-west, with every plugin having their own specific output tree structure (plus some plugins that make no outputs). In particular, plugins with non-deterministic output paths (without parsing the file) are a challenge to make work with Bazel, which expects to know every path. Using declare_directory is generally not a viable solution, as a lot of language rules do not support it for srcs.
Dependencies: Due to the cross cutting nature, a ruleset that supports many languages also ends up with many dependencies, which may end up locking in end-users to versions they do not want and makes ruleset maintenance challenging. The arrival of bzlmod may alleviate this pain by shifting the entire dependency problem onto bzlmod.

@pcj's talk at Bazelcon sums up a lot of the difficulty in this domain and introduces the Gazelle solution.

What's the Current State?

The top level dependency for all proto rules is bazelbuild/rules_proto, which wraps the native implementations of proto_library etc and is in the process of shifting that to Starlark. This does the hard work of walking the tree of deps to assemble the proto descriptors and providers that are used to drive protoc for language specific rules.

For this, there are two main options that support a variety of languages:

stackb/rules_proto: v2 was recently released that solves some of the above difficulties using a Gazelle plugin.
rules_proto_grpc: A fork of v1 rules_proto that uses only Starlark rules.

See also the history section here for a good summary.

Neither of these is going to solve all the problems for everybody, making them somewhat complimentary, since there is always going to be a split of users wanting to use Gazelle vs not. Additionally, both don't and won't be able to have out-of-the-box support every lang/plugin/use case; for more obscure uses, users will have to write some Gazelle or Starlark.

In my perspective, workspaces with lots of proto are going to benefit from Gazelle; conversely, workspaces with less proto are perhaps going to want to avoid pulling in Gazelle and prefer to use the Starlark version. Hence the 'complementary' note above and a migration path between the two would perhaps be beneficial.

Beyond these two, a number of language specific rulesets implement proto/grpc rules themselves (e.g rules_go) and additionally the protobuf/grpc repos have built-in support for a handful of languages. For users that are only using that single language, these are the obvious best choice.

A point raised during discussion was that a lot of users have custom plugins and end up reinventing the entire protoc invocation. Both repos above have options for attaching custom plugins, so I would be interested to hear where these solutions are lacking:

Pain Points

There are some common issues on proto/grpc that are widely frustrating to users:

Caching: The way protobuf is built with Bazel is incredibly fragile to cache misses, leading to frequent rebuilds of protoc for no clear reason. There is not yet a determined cause, but it could be at least partially related to use_default_shell_env leads to spurious rebuilds protocolbuffers/protobuf#6886. That is not the whole solution though, as a static env still leads to cache misses. This has led to protoc being downloaded as a binary in bazelbuild/rules_proto, but this breaks when a user wants to use a newer protobuf version.
Toolchains: The grpc Bazel build completely bypasses the Python toolchains from rules_python. This leads to the build Python plugin not being compatible with the Python a user is using in their workspace.
Need more input here? What are other people's pain points?

Next Steps

Find Rulesets: Gather some info on the available rulesets for users for proto/grpc. I can start a doc that people can drop ruleset names/URLs in.
Documentation: Using the above collected info, put together some guidance on how a user can pick the best ruleset option for their use case. Bear in mind, there is no single right answer for more advanced use cases, so in some situations we can only provide a reduced set of options.
Examples: Having some examples of more complex use cases from some of the Bazel consultancies to put in the bazelbuild/examples was a suggestion by @aiuto
Fix Upstream: Figure out why Protobuf is so fragile to rebuilds and try to fix that. This will probably require help from the Protobuf team. Similarly for gRPC repo, try solve the toolchain issue above.
Blog Post: Somewhat ephemeral and fast out-of-date, so perhaps defer this until we have good guidance documenation available.
More?