RFC: "V2": strongly‐typed schemas, optional client‐side validation

[scottrippey]

Request For Comments

We are considering implementing a new Version 2 of GroqD, based on the PoC work done in #201, #202.

Goals

Currently, GroqD is unaware of the Sanity Scheam; it only validates the results.
It's up to the developer to know what fields can be queried, what their types are, and how to navigate nested structures.

The goal of this new version is to improve the developer workflow. We want GroqD to be aware of the Sanity Schema, so that it can provide developers with code completion, type-inferrance and type-checking for all queries.

As a side-effect, "client-side validation" becomes redundant, so we'd like to make it optional. This will reduce bundle size and improve performance.

Implementation

Schema Generation

The @sanity-typed/types package can be used to infer strong TypeScript types from a Sanity schema.
However, for most scenarios, the Sanity schema will be defined in a separate project/repo than the client that is using GroqD. So, it will usually be necessary to compile the sanity schema into a .d.ts file, and then copy that into the client project.

Unfortunately, the @sanity-typed/types package is VERY slow to infer the schema. Even if we used the TypeScript compiler to generate the .d.ts file, it would still need to infer the types, which causes major compiler issues and IDE issues.
In order for this solution to be realistic and achievable, we need to solve this problem first.

One possible solution is to create a CLI tool that can be used to generate the required "flattened" type definition from the Sanity schema. This would be very similar to graphql-codegen.

Another solution would be to allow developers to manually specify the types, by hand-editing each document type. While this could be prone to errors, it would allow developers to strongly-type their schema, and could be a worthwhile tradeoff.

Query Builder

The majority of GroqD's API will be intact, but now with strong types. Eg. q.order('_createdAt', 'desc') or q.filterByType("product").

A few of the important API changes:

• The .grab function will be replaced by .projection, which mostly shares the same API, but with some important improvements:
  • You can query fields by simply passing true, like .projection({ name: true }). Result types are inferred from the source schema.
  • .grabOne("name") will become .projection("name"), since this can be overloaded.
• For client-side validation, instead of passing true you can pass a "parser" function, like .projection({ name: (n) => n.toUpperCase() }).
  • The parser is allowed to throw runtime errors, or can cast/parse to a different output type. Like .projection({ name: (n) => n.split(' ') })
  • The parser can be a Zod parser, like .projection({ name: zod.string() })

Additional Considerations

• Syntax should be similar to current GroqD - chainable, immutable, composable
• Some method names might be changed to better match GROQ terminology
  • Eg. projection instead of grab
• Predominately TypeScript, but with loosely-typed strings when necessary
  • We will provide an "escape hatch" for manually specifying types with string queries
  • Perhaps we should have a naming convention to identify loose-types, like filter$(string) vs filter(TypedExpression)
• We might want to give this new version a new name, like groq-builder

[ringods]

@scottrippey regarding InferResultType from the current groq-builder. For example, take this example:

export type Posts = InferResultType<typeof postsQuery>

export async function getAllPosts(): Promise<Posts> {
    return runQuery(postsQuery)
}

The Posts type is strongly typed as { ... }[], so always the array type. When using Typescript in (very) strict mode, I sometimes need to configure a function signature with a type, but I would like the singular type Post.

Would it be possible to change InferResultType to return the singular type? In such a case, my example would be updated to:

export type Post = InferResultType<typeof postsQuery>

export async function getAllPosts(): Promise<Post[]> {
    return runQuery(postsQuery)
}

And I can use the Post type very easy further on.

[scottrippey]

I added (in #248) a InferResultItem<typeof postsQuery> utility, which returns a single item from the result.

[scottrippey]

FYI, we recently implemented "fragments" too, which are really useful for these scenarios. For example:

const postFragment = q.fragment<PostDocument>().project({
  title: q.string(), 
  slug: ["slug.current", q.string()],
});
type PostResult = InferFragmentType<typeof postFragment>;

const postsQuery = q.filterByType('post').project({
  ...postFragment,
  content: q.string(),
});

[oybek-daniyarov]

i recently started using groqd and i love it, this fragment is exactly what i was looking for.

[saiichihashimoto]

Hi! 👋 sanity-typed author here! Love this project and, for context, I was discussing a context-aware groq builder with @judofyr and I think it makes more sense as part of this library, and then I found this discussion!

@sanity-typed/types was very slow but I think it's much better now. Correct me if I'm wrong. But also, I generally think they're separate problems: whether it's through @sanity-typed/types, some codegen, a combination, or just manual typing, I think you're trying to provide the builder with the document types so it can be schema aware.

I think this may be less of a lift than you're imagining. If you changed q.someOperation()... into q<SanityDocuments>().someOperation()..., then you can carry forward the possible types into your operations. I believe you're already building the type of a query as is, but now you can intersect with the sanity documents to narrow some parameters. I'm definitely oversimplifying, but I believe it really boils down to "if each operation new about the sanity document types, what would I narrow this argument to?". That way, you'd still end up with runtime validation AND a schema aware builder for it, which is solid. The "client-side validation" might be redundant, but I can tell you first-hand that everyone still really appreciates it (the number of requests I've received for it is non-trivial).

[saiichihashimoto]

OK I've been using this and... I've decided to integrate it directly into @sanity-typed/client. This is very cool.

[scottrippey]

Glad to hear your excitement!
I actually do have a feature request, I'll create a new discussion for it in your repo!

[saiichihashimoto]

Ok! FYI I've done away with discussions, so just make an issue instead.

[scottrippey]

saiichihashimoto/sanity-typed#661

[saiichihashimoto]

OK, last comment 😉

https://www.npmjs.com/package/@sanity-typed/client#usage-with-groqd-actually-groq-builder

You can refer to it from your "Automatically generating your Sanity Schema" section about the preferred way to use @sanity-typed/client and groq-builder together!

[saiichihashimoto]

BTW, if you want a disgusting number of tests for GROQ:

https://github.com/saiichihashimoto/sanity-typed/tree/main/packages/groq/src

Those should be adaptable for groq-builder. I went through every section of the GROQ spec so it should theoretically cover everything possible. Some people from the @sanity-io team reached out to tell me they actually improved their docs and tools by how covered it is 🤣. If you want any help adapting those tests over to groq-builder, I'd be happy to help.

Also, I made a package for the test-utils you copied over a while ago, so you can get any updates I get from it.

[leighkendell]

Hey hey, love what you are doing with this library and even more looking forward to tighter integration with schemas. Just curious about the current status of "groq-builder", I couldn't see any specific mention outside of this discussion. Is there a roadmap for a 1.0 release? Either way I'm going to have a play on some non-prod projects :D

[scottrippey]

We do not have a formal roadmap, sorry.
Our latest version 0.9.0 is what we consider our "Release Candidate", and the API is stable. We'd be happy for any testing and/or feedback you have!
Our biggest task before releasing is to update our documentation and playground. Once that's ready, we'll move forward with the 1.0 release.

[leighkendell]

Cheers @scottrippey sounds great, I'll give it a go!