Implement predefined field constraints #246

jchadwick-buf · 2024-09-04T00:01:59Z

Renames buf.validate.priv.FieldConstraints to buf.validate.PredefinedConstraint
Renames buf.validate.priv.field to buf.validate.predefined
Removes package buf.validate.priv to buf.validate
Merges buf/validate/expression.proto to buf/validate/validate.proto
Removes //proto/protovalidate/buf/validate:expression_proto
Switches buf/validate/validate.proto from proto3 syntax to proto2 syntax to enable usage of extensions
Adds extension 1000 to max to each ...Rules message
Adds test cases for valid and invalid predefined constraints of every possible rule type for both proto2 and protobuf edition 2023
Added the CEL constant rule that can be used by predefined constraints to refer to themselves specifically.

A predefined constraint in this schema looks like this:

edition = "2023";

package example.v1;

import "buf/validate/validate.proto";

extend buf.validate.StringRules {
  bool valid_path = 1162 [
    (buf.validate.predefined).cel = {
      id: "string.valid_path"
      expression: "!rule && !this.matches('^([^/.][^/]?|[^/][^/.]|[^/]{3,})(/([^/.][^/]?|[^/][^/.]|[^/]{3,}))*$') ? 'not a valid path: `%s`'.format([this]) : ''"
    }
  ];
}

message File {
  string path = 1 [(buf.validate.field).string.(valid_path) = true];
}

This is a breaking change.

… shared-field-rules

jchadwick-buf · 2024-09-06T13:16:12Z

After merging main back in, it looks like there is a new lint error:

Error: Enum option "allow_alias" on enum "Ignore" must be false.

After some inspection it appears this may have to do with Buf v1.40.0, as the same code seems to have been passing in Buf v1.39. Further inspection may be needed. (In any case, it doesn't have anything to do with the changes in this PR as best as I can ascertain.)

proto/protovalidate/buf/validate/validate.proto

docs/shared-constraints.md

tools/protovalidate-conformance/internal/cases/cases_shared.go

see comments

oliversun9 · 2024-09-06T16:39:38Z

After merging main back in, it looks like there is a new lint error:
Error: Enum option "allow_alias" on enum "Ignore" must be false.
After some inspection it appears this may have to do with Buf v1.40.0, as the same code seems to have been passing in Buf v1.39. Further inspection may be needed. (In any case, it doesn't have anything to do with the changes in this PR as best as I can ascertain.)

I think this is a CLI bug, I will take a look.

rodaine · 2024-09-06T18:38:15Z

@oliversun9 @jchadwick-buf once the CLI issue is patched (and released) let's get that in here (or a separate pr that's merged in) as well as fixing the deprecation warnings. I'll merge bypass once the only remaining issue is the breaking change detection.

oliversun9 · 2024-09-06T19:06:33Z

@oliversun9 @jchadwick-buf once the CLI issue is patched (and released) let's get that in here (or a separate pr that's merged in) as well as fixing the deprecation warnings. I'll merge bypass once the only remaining issue is the breaking change detection.

I have a CLI PR up for the fix, but one thing you could do to avoid waiting on the CLI fix to land is to break the ignore comment into two lines, i.e.

// buf:lint:ignore ENUM_NO_ALLOW_ALIAS
// allowance for deprecations. TODO: remove pre-v1.

jchadwick-buf · 2024-09-09T15:11:23Z

@oliversun9 Thanks for the quick fix! It worked, so no need to workaround this. (I did contemplate doing the workaround but it's probably for the best to not merge a PR this big on a Friday afternoon anyways.) I'll try to get all of the ducks in a row here and if there are no objections I'll get this merged and get the runtime PRs under review.

There's definitely more stuff that people will want (rules on arbitrary types, shared message constraints, recursively applying constraints) but I think this is a good opportunity to launch-and-iterate.

Thanks for all of the help, feedback, reviews, etc.!

bufdev · 2024-09-11T18:21:19Z

docs/shared-constraints.md

+>
+> Extension numbers may be from 1000 to 536870911, inclusive. Values from 1000
+> to 49999 are reserved for [Protobuf Global Extension Registry][1] entries, and
+> values from 50000 to 536870911 are reserved for randomly-generated integers.


This isn't actually true. 50000-99999 are reserved for internal use for organizations. The random number thing is actually a somewhat-uncomfortable recommendation for me - I get the birthday paradox, but I'm not thrilled that this is what we're recommending people do. I would prefer saying "choose a number that won't conflict" and leave this up to the user.

Thanks, I was mistaken on the ranges somehow, that's what I get for not actually checking.

I agree we shouldn't recommend it, so I've adjusted the documentation altogether:

Fixed the ranges.

No longer explicitly recommends randomly-generated integers

Discourages use of 100000....536870911 for publicly-consumed schemas at all, due to risk of conflicts

But keeping a suggestion from Miguel, I kept a note that suggests using a high-quality random source if the user decides to use a randomly generated integer anyway, under the belief that it's better if they do that versus merely choose arbitrary numbers.

WDYT?

Remove random number generation references

Coalesce extension ranges

bufdev · 2024-09-11T18:21:59Z

docs/shared-constraints.md

+`buf.validate.FloatRules`, as follows:
+
+```proto
+import "buf/validate/shared/constraints.proto";


Note that these docs no longer match impl

Documentation has been updated.

bufdev · 2024-09-11T18:24:39Z

proto/protovalidate/buf/validate/validate.proto

+// `Violations` is a collection of `Violation` messages. This message type is returned by
+// protovalidate when a proto message fails to meet the requirements set by the `Constraint` validation rules.
+// Each individual violation is represented by a `Violation` message.
+message Violations {


I wouldn't expect these to be intermixed in order with the *Constraints definitions. These should likely be at the bottom of the file

Moved Violations and Violation to end of file.

bufdev · 2024-09-16T15:50:37Z

proto/protovalidate/buf/validate/validate.proto

+  // constraints that can then/ be set on field options of other messages to
+  // apply reusable field constraints.
+  //
+  // When using randomly generated numbers, please use a high-quality source of


I still don't understand why we are referencing random number generation in the docs - I've read the comments, but this doesn't seem like a good idea.

OK, references to random number generation are just removed.

bufdev · 2024-09-16T15:51:12Z

proto/protovalidate/buf/validate/validate.proto

+  // messages to apply reusable field constraints.
+  //
+  // [1]: https://github.com/protocolbuffers/protobuf/blob/main/docs/options.md
+  extensions 1000 to 99999;


I don't understand why we're splitting up the extension ranges into two sections - I've never seen that before. In the absence of a strong reason, coalesce.

Was suggested here and seemed reasonable enough to me. Either way, it's coalesced again now.

… shared-field-rules

rodaine · 2024-09-17T15:23:07Z

Thanks @jchadwick-buf! Herculean effort.

Like protovalidate-go and protovalidate-java, we need to adjust the code to handle dynamic descriptor sets more robustly, since we need to jump between resolving the protovalidate standard rules and the predefined rule extensions. This necessitates adding a couple of additions to the API surface, namely `ValidatorFactory::SetMessageFactory` and `ValidatorFactory::SetAllowUnknownFields`, which controls instantiation of unknown dynamic types and whether or not to ignore unresolved rules, respectively. Like other protovalidate runtimes, we will default to failing compilation when unknown predefined rules are encountered. This should not break existing users but will prevent silent incorrect behavior. TODO: - [x] Skip reparse when there are no empty fields—this way we can avoid pessimizing the common case - [x] Add an option to fail when unknown rule fields are unable to be resolved. - [x] Update for protobuf changes in bufbuild/protovalidate#246. This will depend on bufbuild/protovalidate#246.

This depends on bufbuild/protovalidate#246.

- Adds the ability to specify an `ExtensionRegistry` _and_ a `TypeRegistry` for resolving protobuf messages. Ordinarily, only a `TypeRegistry` would necessarily be needed. However, we need to be able to resolve extensions defined in file descriptor sets we don't control, which means we need to be able to reparse _to and from_ the user's descriptors in the worst case: _to_ the user's descriptors to get the extended rule message (whose message type descriptors may have a different hashcode and thus may not resolve using just an `ExtensionRegistry` alone) and back _from_ the user's descriptors in order to parse the `priv`/`shared` field. - Refactors some of the code around reparsing and extensions in general: - Reparsing options for protovalidate built-ins will always use a static extension registry. - Adds the `rule` variable. - Some refactoring is done around the individual rule compilation, since the code was getting a bit unwieldy. - Updates the conformance runner to generate an `ExtensionRegistry` and a `TypeRegistry`. This enables the conformance runner to pass both the old conformance test suite and the new one, regardless of whether the proto descriptors match up. TODO: - [x] Update to new version of protovalidate protos when they are merged. This will depend on bufbuild/protovalidate#246.

The code generator for the php language doesn't work with `proto2`. We were able to generate php code required in the SDK until [protovalidate](https://github.com/bufbuild/protovalidate) migrated from `proto3` to `proto2` in bufbuild/protovalidate#246. Communicating with the [buf team](https://github.com/bufbuild), I've learned a new feature `exclude_types` to be introduced which basically excludes any types you specified with it. Updated `buf.gen.yaml` with `exclude_types`: ```yaml version: v2 plugins: - remote: buf.build/protocolbuffers/php:v30.2 out: ./src exclude_types: - buf.validate - remote: buf.build/grpc/php:v1.71.0 out: ./src exclude_types: - buf.validate ``` Also this PR updates some dependencies and introduces support for `php 8.4` while removing support for `php 8.3`. Signed-off-by: Oğuzhan Durgun <oguzhandurgun95@gmail.com>

Implement shared field rules

0d5bfe9

jchadwick-buf mentioned this pull request Sep 4, 2024

Implement predefined field constraints bufbuild/protovalidate-go#144

Merged

jchadwick-buf added 6 commits September 3, 2024 20:15

Fix bad formatting

8657cd2

Add shared rule tests for each type

bf72afb

Add tests for 'rule' constant

9030e64

Add tests using shared rules from proto3

0236de1

Add docs for shared field rules

d72b7d7

Further documentation improvements

4ef8796

jchadwick-buf changed the title ~~Implement shared field rules~~ Implement shared field constraints Sep 4, 2024

jchadwick-buf requested a review from rodaine September 4, 2024 05:50

jchadwick-buf mentioned this pull request Sep 4, 2024

Implement shared field constraints bufbuild/protovalidate-python#186

Closed

rodaine added Feature New feature or request Breaking Change Describes a breaking change to the protovalidate API labels Sep 4, 2024

rodaine linked an issue Sep 4, 2024 that may be closed by this pull request

[BREAKING CHANGE] Reusing Custom Validation Rules #51

Closed

Merge branch 'main' of https://github.com/bufbuild/protovalidate into…

13aee7a

… shared-field-rules

jchadwick-buf mentioned this pull request Sep 4, 2024

Add support for shared field constraints bufbuild/protovalidate-cc#60

Closed

2 tasks

Fix signed/unsigned confusion in shared rules test

09dcec8

jchadwick-buf mentioned this pull request Sep 6, 2024

Implement shared field constraints bufbuild/protovalidate-java#176

Closed

Merge branch 'main' of https://github.com/bufbuild/protovalidate into…

23b00c8

… shared-field-rules

rodaine previously approved these changes Sep 6, 2024

View reviewed changes

proto/protovalidate/buf/validate/validate.proto Show resolved Hide resolved

docs/shared-constraints.md Outdated Show resolved Hide resolved

tools/protovalidate-conformance/internal/cases/cases_shared.go Outdated Show resolved Hide resolved

jchadwick-buf added 3 commits September 6, 2024 14:07

Documentation improvements

74e3f64

Add test cases covering mixed shared and custom constraints

bb9336f

Add test with standard, shared and custom constraints on field

006e5d5

rodaine approved these changes Sep 6, 2024

View reviewed changes

bufdev reviewed Sep 11, 2024

View reviewed changes

Shared -> predefined, fix docs, gen, etc.

95e8a98

jchadwick-buf changed the title ~~Implement shared field constraints~~ Implement predefined field constraints Sep 11, 2024

This was referenced Sep 12, 2024

Implement predefined field constraints bufbuild/protovalidate-cc#61

Merged

Implement predefined field constraints bufbuild/protovalidate-java#178

Merged

Implement predefined field constraints bufbuild/protovalidate-python#189

Merged

oliversun9 mentioned this pull request Sep 13, 2024

Expand buf lint PROTOVALIDATE rule to lint example and predefined rules bufbuild/buf#3317

Merged

bufdev reviewed Sep 16, 2024

View reviewed changes

jchadwick-buf added 2 commits September 16, 2024 12:25

updates

b1890fc

generate

eef5323

jchadwick-buf requested a review from bufdev September 16, 2024 17:41

bufdev approved these changes Sep 16, 2024

View reviewed changes

jchadwick-buf added 2 commits September 17, 2024 09:26

oops, no need for a helper here

c1862ea

Merge branch 'main' of https://github.com/bufbuild/protovalidate into…

444deab

… shared-field-rules

jchadwick-buf merged commit 81eafa5 into bufbuild:main Sep 17, 2024
3 of 4 checks passed

jchadwick-buf deleted the shared-field-rules branch September 17, 2024 15:24

migmartri mentioned this pull request Sep 17, 2024

reuse name validation role in our protos chainloop-dev/chainloop#1328

Closed

jchadwick-buf added a commit to bufbuild/protovalidate-python that referenced this pull request Sep 25, 2024

Implement predefined field constraints (#189)

3e2e3e3

This depends on bufbuild/protovalidate#246.

dropwhile mentioned this pull request Sep 28, 2024

Add support for predefined constraints sudorandom/protoc-gen-connect-openapi#35

Open

jakedoublev mentioned this pull request Oct 16, 2024

fix(policy): enhance proto validation across policy requests opentdf/platform#1656

Merged

oguzhand95 mentioned this pull request Feb 27, 2025

[Question] Can't generate PHP code #319

Closed

timostamm mentioned this pull request Mar 25, 2025

Implement predefined constraints bufbuild/cel-es#83

Merged

oguzhand95 mentioned this pull request Apr 8, 2025

chore(deps): Update dependencies cerbos/cerbos-sdk-php#112

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement predefined field constraints #246

Implement predefined field constraints #246

jchadwick-buf commented Sep 4, 2024 •

edited

Loading

jchadwick-buf commented Sep 6, 2024

oliversun9 commented Sep 6, 2024

rodaine commented Sep 6, 2024 •

edited

Loading

oliversun9 commented Sep 6, 2024

jchadwick-buf commented Sep 9, 2024 •

edited

Loading

bufdev Sep 11, 2024

jchadwick-buf Sep 11, 2024

bufdev Sep 16, 2024

jchadwick-buf Sep 16, 2024

bufdev Sep 11, 2024

jchadwick-buf Sep 11, 2024

bufdev Sep 11, 2024

jchadwick-buf Sep 11, 2024

bufdev Sep 16, 2024

jchadwick-buf Sep 16, 2024

bufdev Sep 16, 2024

jchadwick-buf Sep 16, 2024

rodaine commented Sep 17, 2024

Implement predefined field constraints #246

Implement predefined field constraints #246

Conversation

jchadwick-buf commented Sep 4, 2024 • edited Loading

jchadwick-buf commented Sep 6, 2024

oliversun9 commented Sep 6, 2024

rodaine commented Sep 6, 2024 • edited Loading

oliversun9 commented Sep 6, 2024

jchadwick-buf commented Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rodaine commented Sep 17, 2024

jchadwick-buf commented Sep 4, 2024 •

edited

Loading

rodaine commented Sep 6, 2024 •

edited

Loading

jchadwick-buf commented Sep 9, 2024 •

edited

Loading