GraphQL Conformance

View the dashboard

What is this?

This project tests whether open-source GraphQL implementations behave the same way. It runs the same queries against each implementation and compares the results to a reference (graphql-js).

Implementations tested:

• graphql-js 16 (reference)
• graphql-js 17
• graphql-java
• Viaduct
• graphql-go
• gqlgen
• Hot Chocolate
• graphql-dotnet
• async-graphql
• Juniper

How it works

Each implementation is wrapped in a small harness that accepts a schema file and a query file, builds the schema, runs the query, and prints the result as JSON. All harnesses use the same deterministic resolvers (the Wiring Spec) so that the only differences come from the GraphQL engine itself.

Test cases are generated randomly using the Viaduct Arbitrary toolkit. This produces arbitrary GraphQL schemas, documents, and variables, which are stored in corpus.

A coordinator runs every test case against every implementation, compares outputs to the reference, and records the results.

Requirements

• mise — manages tool versions (Node.js, Go, Java, .NET, Rust)

That's it. mise handles installing the right versions of everything else.

Quick start

make build          # clone libraries and build all implementations
make test           # run all tests
make run-conformer  # run conformance suite and update the dashboard
make serve-site     # serve the dashboard locally

Other commands

make gen-corpus                                        # regenerate test cases
make run-impl IMPL=graphql-go TEST=corpus/0/0          # run one impl on one test
make diff-impl IMPL=graphql-go TEST=corpus/0/0         # diff an impl against the reference
make clean-corpus                                      # delete generated test cases (keeps corpus/0)
make clean-results                                     # delete stored results
make clean                                             # clean all build artifacts

Project layout

corpus/           test cases (schema + query + optional variables)
corpus-gen/       test case generator (Kotlin)
conformer/        coordinator that runs tests and compares results (Node.js)
impls/            one directory per GraphQL implementation
results/          results store (writes to results/data/)
site/             static dashboard (reads from site/data/)

Adding an implementation

1. Create impls/<name>/ with code that implements the Wiring Spec
2. Add a Makefile with build, test, clean targets
3. Add an entry to config.json
4. Add any new tool versions to .mise.toml
5. Run make build && make test