BLAKE2b gadget #1767

boray · 2024-07-23T17:29:09Z

Closes #1754

Ingredients:

BLAKE2b hash function implementation with arbitrary digest length https://www.blake2.net/blake2.pdf
divMod64 and addMod64 needed for compression function
Gadgets.or()
UInt64.or()
Add zero and one accessors to UInt8

…e/blake2b

src/lib/provable/gadgets/bitwise.ts

Co-authored-by: Martin Minkov <minkovlmartin@gmail.com>

Trivo25

Great job! I left some initial comments after a first pass

src/examples/crypto/blake2b/blake2b.ts

src/lib/provable/crypto/hash.ts

src/lib/provable/gadgets/blake2b.ts

src/lib/provable/gadgets/gadgets.ts

Trivo25 · 2024-07-29T07:36:14Z

src/lib/provable/test/blake2b.unit-test.ts

+}
+
+
+function testVectors() {


where are some of these test vectors from? we should use some official ones also (if they arent the official ones!)

Official test vectors have key parameter which I didn't implement. Therefore, I had to create test vectors manually.

I would in any case include tests with longer preimages, to make sure the code works when two blocks are required (meaning t1 is nonzero). I believe the longer preimage in these tests was 43 bytes. Let's try with 128 for example.

Therefore, I had to create test vectors manually.

How?

tests/vk-regression/vk-regression.json

mitschabaude · 2024-07-30T09:53:20Z

@boray since this relies on the changes in #1763, you should change the base branch to feature/divmod32-quotientbits-bound to make this easier to review

mitschabaude · 2024-07-30T09:58:15Z

I changed the base branch, which reveals that you didn't merge that branch but added it as a single squashed commit.

I highly recommend to never do that! it means we have the same changes in two different commits, which git can't resolve and instead will mark them as conflicts. (see "This branch has conflicts that must be resolved" on this PR now)

the preferred way to use git at o1Labs is to basically always join branches using git merge, which reuses the original commits and so doesn't create any fake conflicts

…und' into feature/blake2b

mitschabaude

a few quick comments on the gadgets additions!

src/lib/provable/gadgets/arithmetic.ts

mitschabaude · 2024-08-22T14:05:32Z

src/lib/provable/gadgets/arithmetic.ts

+  if (quotientBits === 1) {
+    quotient.assertBool();
+  } else {
+    rangeCheckN(quotientBits, quotient);
+  }


for efficiency this should use rangeCheck64() instead of rangeCheckN() if quotientBits === 64!

quotientBits is not necessarily 64 since it's calculated from nBits. So, I think it's better to keep rangeCheckN().

I suggested to use rangeCheck64 if quotientBits === 64. so what I mean is, test for that condition, and use the efficient range check in that case

Okay, got it. This improvement is relevant for divMod32 too!

no it's not! there is no dedicated single-gate 32-bit range check, like there is for 64 bits

I didn't know that! I removed divMod32 change with the latest commit. AFAIU from the kimchi page of the proof systems book, only 64-bit range check has a dedicated single-gate. Quoting from the book:

The RangeCheck0 gate can also be used on its own to perform 64-bit range checks by constraining witness cells 1-2 to zero.

It is true that only range checks for 64bits have a dedicated Kimchi gate. However, one can also perform 32bit range checks just less efficiently. Maybe let's not remove the o1js function for it as it might be better from a ux point of view, instead of asking the zkapp developer to use a rc64 for x and another one for x * 2^32

For 32 bits there are two different ways to do it in 2.5 gates, and one of them is already used -- the one that doesn't use the range check gate.
Using that one is better because the range check gate and lookup table are only optionally included in the proof system, and the proof gets slower by including them. So it only pays off if you need a lot of range checks.

src/lib/provable/gadgets/bitwise.ts

querolita

Great PR 😄

For today I am leaving some comments regarding the arithmetic side of this PR. Tomorrow I will look deeper into the actual hash function and leave feedback there as well.

src/lib/provable/gadgets/gadgets.ts

querolita · 2024-08-28T16:32:24Z

src/lib/provable/gadgets/gadgets.ts

+  divMod64,
+
+  /**
+   * Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128] and returns the result modulo 2^64.


Suggested change

* Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128] and returns the result modulo 2^64.

* Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128) and returns the result modulo 2^64.

TBH, I am not sure which one is right. The ranges in addMod32 comment block doesn't match too. I took the comment for addMod64 from addMod32 since it's the exact same function but modulo 2^64.

first of all, all ranges are always of the form [a, b) i.e. open on the upper end

second, 128 bits is wrong. it uses a 1-bit range check on the quotient. for that to work, the addition result has to be at most 65 bits

also, you say further below that "the gadget assumes both inputs to be in the range [0, 2^64)". that will definitely work and is the intended use case, so stick with that here in the description as well

src/lib/provable/gadgets/gadgets.ts

querolita · 2024-08-28T17:14:45Z

src/lib/provable/test/bitwise.unit-test.ts

+  (x, y) => {
+    if (x >= 2n ** 64n || y >= 2n ** 64n)
+      throw Error('Does not fit into 64 bits');
+    return x & y;


why do you return & in this case but Bitwise.or in the next?

Good catch! That should be | not &. Interestingly, it doesn't fail the unit test. Any ideas, @mitschabaude?

my only idea is that you use such a small number of runs that with high likelyhood, one of the two inputs is bigger than $2^{64}$ in every single run, so you get an error every time.

try it with uint(64) as the random generator instead of maybeField!

I think you can actually remove this unit test that creates a proof, and stick with the ones above that only run a circuit and check constraints

The proof one will make CI slower and provides hardly any additional value

While developing smart contracts, sometimes it works without proofs but fails with them. Isn't it beneficial to test with proofs. Is it different for primitives?

It's different here yeah, for various reasons.

For one, we are already testing many extremely similar methods with proofs in this file, the additional surface covered is funny

src/lib/provable/int.ts

querolita · 2024-08-28T17:28:19Z

src/lib/provable/gadgets/arithmetic.ts

+function divMod64(n: Field, nBits = 128) {
+  assert(
+    nBits >= 0 && nBits < 255,
+    `nBits must be in the range [0, 255), got ${nBits}`


Is 255 not included because of Pasta curves?

I am not sure why we don't include 255. I followed the range checking practice used in divMod32. Maybe @mitschabaude knows the reason?

Because the field size is less than 2^255, there are 255 bit integers that have two different representations in the field (in fact that's true for almost all 255 bit integers since the field size is just slightly over 2^254)

That non-uniqueness would make the divmod gadget unsound: the prover could pick between two different quotient/remainder splits, one of which is wrong.

src/lib/provable/gadgets/arithmetic.ts

querolita · 2024-08-28T17:37:34Z

src/lib/provable/gadgets/arithmetic.ts

+}
+
+function addMod64(x: Field, y: Field) {
+  return divMod64(x.add(y), 128).remainder;


What's the reasoning behind nBits being 128 in this case? (Instead of a larger or smaller value perhaps). Is it to prevent the quotient from being larger than 64 bits upon reduction?

yeah if both inputs are 64 bits then the result is 65 bits, that would cause an efficient (boolean) check for the quotient

so the question: can't we assume that the inputs are 64 bits and therefore use nBits=65?

yes, it should be 65. It used to be 128 at some point. I adapted the changes made in #1763 and forgot to update nBits.

querolita

Finished my first pass, left some comments and questions 👍🏻

src/lib/provable/gadgets/blake2b.ts

querolita · 2024-08-29T11:35:08Z

src/lib/provable/gadgets/blake2b.ts

+  for (let i = 0; i < input.length; i++) {
+    if (state.buflen === 128) {
+      state.t[0] = state.t[0].add(128);
+      if (state.t[0].equals(UInt64.zero)) {


Can you explain in what situations this conditional would succeed? I am asking because it seems like the counter t[0] gets updated using a normal add, whereas the t[1] uses a modular addition instead. Are you thinking of a UInt64 overflow? If so, maybe it's better to use addMod64 here as well.

btw @boray this is a bug. you can't use a Bool as condition, it's an object, always truthy!

I highly recommend to do out-of-circuit logic on plain JS values, like bigint to avoid mistakes like this, i.e. here I would use

if (state.t[0].toBigint() === 0n) {

ah but wait, is this supposed to be circuit code and state.t[0] is a variable? then you fundamentally can't use a JS if condition anyway

state.t[0] is a variable but I am not sure if that logic should be in-circuit or not. I added the in-circuit version as a comment.

I implemented this part with reference to the snippet below from the BLAKE2B RFC.

ctx->t[0] += ctx->c; // mark last block offset if (ctx->t[0] < ctx->c) // carry overflow ctx->t[1]++; // high word

src/lib/provable/gadgets/blake2b.ts

mitschabaude · 2024-09-11T06:27:04Z

src/lib/provable/gadgets/blake2b.ts

+      UInt64.from(
+        buf[i * 8]
+          .toUInt64()
+          .or(buf[i * 8 + 1].toUInt64().leftShift(8))
+          .or(buf[i * 8 + 2].toUInt64().leftShift(16))
+          .or(buf[i * 8 + 3].toUInt64().leftShift(24))
+          .or(buf[i * 8 + 4].toUInt64().leftShift(32))
+          .or(buf[i * 8 + 5].toUInt64().leftShift(40))
+          .or(buf[i * 8 + 6].toUInt64().leftShift(48))
+          .or(buf[i * 8 + 7].toUInt64().leftShift(56))


this should be done with addition and multiplication instead of or() and leftShift()

e.g.

- .or(buf[i * 8 + 1].toUInt64().leftShift(8)) + .add(buf[i * 8 + 1].value.mul(1n << 8n))

in a circuit, addition and multiplication is cheap while bitwise operations are expensive!

you can also leave it and we do a second pass for efficiency, and you focus on correctness for now?

I wasn't aware that add/mul is cheaper than bitwise ops. Let's focus on correctness now and leave this improvement to the efficiency pass.

Basically, as a rule of thumb, binary operations which are cheap on CPU are expensive in cryptography (where arithmetic operations in a field are the "cheap" operations instead). Modeling binary operations in a circuit becomes a big overhead, so anytime where binary ops can be replaced with arithmetic counterparts, is preferred for efficiency reasons.

Either way, let's not forget that second pass for efficiency in a later PR.

@boray you can think of add(), mul() and assertEquals() as the primitive operations out of which everything else is built. nothing else is as efficient, and most things are far less efficient

in the end, all of this turns into polynomial equations, and polynomials are made up of addition and multiplication

I refactored in the last commit!

querolita

Leaving two more comments before acceptance. These are related to the actual constraints being created behind the scenes.

querolita · 2024-09-19T14:02:50Z

src/lib/provable/gadgets/arithmetic.ts

+    let nBigInt = n.toBigInt();
+    let q = nBigInt >> 64n;
+    let r = nBigInt - (q << 64n);
+    return {


I believe that when n is a constant, there are no constraints being created (the fields are returned before the rangechecks and division constraint are called). Is that the desired behavior?

Yes! For constant cases (where all variables of a computation are constant), we calculate the result without creating constraints and return another constant value

querolita · 2024-09-19T17:00:20Z

src/lib/provable/gadgets/blake2b.ts

+  /*
+  state.t[1] = state.t[1].add(
+    Provable.if(
+      state.t[0].lessThan(UInt64.from(state.buflen)),


If this needs to go in the circuit, then the same lines of the update part should also be written in circuit manner. In any case, would it make sense to have a final constrain checking t[1]*2^64+t[0]=total_length? That, together with the modular additions gives you a correct decomposition.

Now, going back to what @mitschabaude said about the state.t[0] being a variable, one cannot decide the design of the circuit depending on the value of a witness cell. The circuit must have the same "universal shape", regardless of the concrete values of the variables.

So putting all the above together, it's not a matter of whether that logic needs to go into the circuit or not. It's about not being able to decide upon the concrete value a variable takes. Thus, I believe this part should be rewritten, perhaps with the use of Provable.if (but I might be missing the right o1js know-how, so maybe ask another team member for further insights).

At least in Kimchi, one would create a circuit that is already "bifurcated". Meaning, all the constraints hold and they are the same, no matter what concrete values are passed to the prover. So we play with multiplications to "select" one branch or another, but both sides should eventually evaluate to 0 to satisfy the constraint system.

mitschabaude · 2024-09-20T13:44:07Z

src/lib/provable/test/blake2b.unit-test.ts

+for (let { digest_length ,preimage, hash } of testVectors()) {
+  let actual = Gadgets.BLAKE2B.hash(Bytes.fromString(preimage), digest_length);
+  expect(actual.toHex()).toEqual(hash);
+}


these tests should run in a circuit!

mitschabaude · 2024-09-20T13:55:00Z

src/lib/provable/gadgets/blake2b.ts

+
+type State = {
+  h: UInt64[];
+  t: UInt64[];


as far as I can tell, t never contains variables, just constants that change depending on the input size (which is also constant)

therefore, it's better to change the type to bigint or number, to avoid confusion with variables (see other comments where we were confused):
Also, nicer to use a fixed-size array since it's just two elements.

Suggested change

t: UInt64[];

t: [bigint, bigint]

mitschabaude · 2024-09-20T13:55:50Z

src/lib/provable/gadgets/blake2b.ts

+  /*
+  state.t[1] = state.t[1].add(
+    Provable.if(
+      state.t[0].lessThan(UInt64.from(state.buflen)),
+      UInt64.one,
+      UInt64.zero
+    )
+  );
+*/


mitschabaude · 2024-09-20T14:06:34Z

src/lib/provable/gadgets/gadgets.ts

+  divMod64,
+
+  /**
+   * Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128] and returns the result modulo 2^64.


first of all, all ranges are always of the form [a, b) i.e. open on the upper end

second, 128 bits is wrong. it uses a 1-bit range check on the quotient. for that to work, the addition result has to be at most 65 bits

also, you say further below that "the gadget assumes both inputs to be in the range [0, 2^64)". that will definitely work and is the intended use case, so stick with that here in the description as well

mitschabaude · 2024-09-20T14:08:27Z

src/lib/provable/test/bitwise.unit-test.ts

+await equivalentAsync({ from: [maybeField, maybeField], to: field }, { runs })(
+  (x, y) => {
+    if (x >= 2n ** 64n || y >= 2n ** 64n)
+      throw Error('Does not fit into 64 bits');
+    return x | y;
+  },
+  async (x, y) => {
+    let proof = await Bitwise.or(x, y);
+    return proof.publicOutput;
+  }
+);
+


I'd say remove this, the in-circuit equivalence tests above are much more thorough and creating a proof here just makes CI slower

mitschabaude · 2024-09-20T14:11:01Z

tests/vk-regression/vk-regression.json

@@ -204,6 +208,10 @@
      "Poseidon": {
        "rows": 208,
        "digest": "afa1f9920f1f657ab015c02f9b2f6c52"
+      },
+      "BLAKE2B": {
+        "rows": 6012,


mitschabaude · 2024-09-20T14:20:04Z

src/lib/provable/gadgets/blake2b.ts

+        state.t[1] = state.t[1].addMod64(UInt64.one); // high word
+      }
+      state = compress(state, false); // compress (not last)
+      state.buflen = 0; // counter to zero


Shouldn't this also reset state.buf to an empty array? If you don't do this, you'll start overwriting buf at the start but the rest of it will still contain the old entries.

and if yes, wouldn't it be less confusing in general to always use state.buf.length instead of an extra parameter buflen, so that they can't get out of sync?

mitschabaude · 2024-09-20T14:22:59Z

src/lib/provable/gadgets/blake2b.ts

+    out[i] = UInt8.from(
+      state.h[i >> 3].rightShift(8 * (i & 7)).and(UInt64.from(0xff))


if you aim for efficiency: here you can also use arithmetic instead of bitwise operations

I tried to replace this line with state.h[i >> 3].div(2 ** (8 * (i & 7))).mod(0xff) but it increased the row number a lot. I am not sure if I'm missing something.

You can check out the bytesToWord function that's used in the keccak and sha2 implementations!
Probably a good idea to use that function

boray added 7 commits July 18, 2024 23:41

add blake2b

d357853

add or gadget and improvements

cb93722

Merge branch 'feature/blake2b' of github.com:o1-labs/o1js into featur…

31abb6e

…e/blake2b

add comments and tests

2846396

fix bitwise test

9dd79aa

add comments to gadgets

24c743a

dump vks

044cbe1

boray marked this pull request as ready for review July 24, 2024 15:19

boray requested review from mitschabaude, Trivo25 and MartinMinkov July 25, 2024 07:58

boray added 4 commits July 26, 2024 03:25

add doccoments and test

81bc5a9

add yoni's fix #1763

58a07a0

Merge remote-tracking branch 'origin/main' into feature/blake2b

72877d7

add UInt32.or()

10836c9

MartinMinkov reviewed Jul 27, 2024

View reviewed changes

src/lib/provable/gadgets/bitwise.ts Outdated Show resolved Hide resolved

boray and others added 2 commits July 28, 2024 15:33

Update src/lib/provable/gadgets/bitwise.ts

00cdb75

Co-authored-by: Martin Minkov <minkovlmartin@gmail.com>

update vk-regression.json

5db6974

boray requested a review from Shigoto-dev19 July 28, 2024 18:08

Trivo25 reviewed Jul 29, 2024

View reviewed changes

boray added 2 commits July 29, 2024 13:12

fixes and styling

63b939c

remove question marks

e6317f1

mitschabaude changed the base branch from main to feature/divmod32-quotientbits-bound July 30, 2024 09:53

boray added 4 commits July 31, 2024 00:13

Merge remote-tracking branch 'origin/feature/divmod32-quotientbits-bo…

95a71d3

…und' into feature/blake2b

optimize g function additions

90daf62

hardcode IV as UInt64

1938405

dump vks

5e2b28a

mitschabaude reviewed Aug 22, 2024

View reviewed changes

boray added 6 commits August 23, 2024 02:29

simplify or

a86a106

fix formatting

c38118e

fix divMod64 range check

5bba739

Merge remote-tracking branch 'origin/v2' into feature/blake2b

e9c5a29

fix elliptic curve to match v2

555070b

fix formatting

d071e7b

querolita reviewed Aug 28, 2024

View reviewed changes

style(gadgets.ts): wrap comments for readability

9b32c4d

querolita reviewed Aug 29, 2024

View reviewed changes

querolita self-assigned this Aug 29, 2024

boray added 13 commits September 5, 2024 16:19

fix(bitwise.unit-test.ts): correct typo

70169e3

perf: optimize divmod rangechecks

acd32bb

fix: remove incorrect optimization

ec5b6a7

fix(arithmetic.ts): fix range assertion

b741027

docs: clarify output description

a34d8c2

fix(blake2b.ts): add range check for digestLength

6ba2968

fix(blake2b.ts): range check input byte length

d841a51

test: add test vector

d999c09

feat: add last block flag and state type

e5bd574

fix: counter logic

1c5f3bd

chore: update reference

7f650b1

docs: add comments

e545799

docs: add more comments

368ae88

mitschabaude reviewed Sep 11, 2024

View reviewed changes

querolita reviewed Sep 19, 2024

View reviewed changes

perf: reduce row number

cb80ccf

mitschabaude requested changes Sep 20, 2024

View reviewed changes

mitschabaude reviewed Sep 20, 2024

View reviewed changes

boray added 2 commits September 20, 2024 22:10

fix: uint64 construction

20a0eac

test: remove in-circuit equivalence tests

cceeef5

	* Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128] and returns the result modulo 2^64.
	* Addition modulo 2^64. The operation adds two {@link Field} elements in the range [0, 2^128) and returns the result modulo 2^64.

		out[i] = UInt8.from(
		state.h[i >> 3].rightShift(8 * (i & 7)).and(UInt64.from(0xff))

BLAKE2b gadget #1767

Are you sure you want to change the base?

BLAKE2b gadget #1767

Conversation

boray commented Jul 23, 2024 • edited Loading

Trivo25 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude commented Jul 30, 2024

mitschabaude commented Jul 30, 2024 • edited Loading

mitschabaude left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

querolita left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

querolita left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

querolita Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

mitschabaude Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

mitschabaude Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

querolita left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boray commented Jul 23, 2024 •

edited

Loading

mitschabaude commented Jul 30, 2024 •

edited

Loading

mitschabaude Sep 20, 2024 •

edited

Loading

mitschabaude Sep 5, 2024 •

edited

Loading

querolita Sep 17, 2024 •

edited

Loading

mitschabaude Sep 20, 2024 •

edited

Loading

mitschabaude Sep 20, 2024 •

edited

Loading

mitschabaude Sep 20, 2024 •

edited

Loading