Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blake3 support #710

Open
AlessandroPatti opened this issue Oct 27, 2023 · 3 comments · May be fixed by #715
Open

Blake3 support #710

AlessandroPatti opened this issue Oct 27, 2023 · 3 comments · May be fixed by #715

Comments

@AlessandroPatti
Copy link
Contributor

Bazel now supports blake3 as a hashing algorithm. It'd be great if we could start using it with bazel-remote as it promises impressive perfomance boosts

@mostynb
Copy link
Collaborator

mostynb commented Oct 30, 2023

The main blocker to adding support for BLAKE3 is that we need to consider if an update to the cache directory format is required, since the current format does not explicitly say what digest function the blobs are stored in.

If it is safe to assume that we will never encounter digest collisions between SHA256 and BLAKE3, then maybe we can keep the current format unchanged. If that's the case then maybe it's OK for bazel-remote to not care what the format of on-disk blobs are.

(Aside: I think someone mentioned that BLAKE3 is only fast on amd64. Something to bear in mind.)

@prestonvanloon
Copy link

prestonvanloon commented Nov 29, 2023

I think it's a safe assumption that BLAKE3 and SHA256 would not have a digest collision.

However, I think there are some assumptions that sha256 is the only hashing algorithm for bazel. Example:

if hash == emptySha256 {

However, there are new fields in the protos that explain which digest function is used, so it would be trivial to add the appropriate paths for BLAKE3.

https://github.com/bazelbuild/remote-apis/blob/6c32c3b917cc5d3cfee680c03179d7552832bb3f/build/bazel/remote/execution/v2/remote_execution.proto#L1555

@adincebic
Copy link

I tried using blake3 with bazel-remote and it works fine when I disable remote cache compression. To be precise, i needed to set storage mode to 'uncompressed' in bazel-remote. However, if I set it to zstd than it stops working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants