Add strict type enforcement for requests & responses

[Timothy-Gonzalez]

Right now, types are a suggestion and we cast for nearly every request. We should have strict typing for requests and responses, and not be writing validation code for types.

• admission
• auth
• event
• mail
• mentor
• newsletter
• notification
• profile
• puzzle
• registration
• s3
• shop
• staff
• user
• version

[Timothy-Gonzalez]

OpenAPI spec seems to be the standard. Also works easily with a bunch of documentation tools like Swagger. Even better, we can defined the spec and convert to ts interfaces using openapi-typescript.

[Timothy-Gonzalez]

Alternative is zod, need to investigate which will work best.

[Timothy-Gonzalez]

Investigated - openapi is the easier to work with, especially if we want to use Swagger. If we use openapi, we can also use it for pretty much any other documentation generation. This will also enable using openapi clients for the other systems teams, so they can have a much easier time using our api (would require them to convert over to an openapi client, though).

An example openapi spec:

Expand

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
  termsOfService: http://swagger.io/terms/
  contact:
    name: Swagger API Team
    email: apiteam@swagger.io
    url: http://swagger.io
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
servers:
  - url: https://petstore.swagger.io/v2
paths:
  /pets:
    get:
      description: |
        Returns all pets from the system that the user has access to
        Nam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.

        Sed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.
      operationId: findPets
      parameters:
        - name: tags
          in: query
          description: tags to filter by
          required: false
          style: form
          schema:
            type: array
            items:
              type: string
        - name: limit
          in: query
          description: maximum number of results to return
          required: false
          schema:
            type: integer
            format: int32
      responses:
        '200':
          description: pet response
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Pet'
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
    post:
      description: Creates a new pet in the store. Duplicates are allowed
      operationId: addPet
      requestBody:
        description: Pet to add to the store
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/NewPet'
      responses:
        '200':
          description: pet response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Pet'
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
  /pets/{id}:
    get:
      description: Returns a user based on a single ID, if the user does not have access to the pet
      operationId: find pet by id
      parameters:
        - name: id
          in: path
          description: ID of pet to fetch
          required: true
          schema:
            type: integer
            format: int64
      responses:
        '200':
          description: pet response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Pet'
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
    delete:
      description: deletes a single pet based on the ID supplied
      operationId: deletePet
      parameters:
        - name: id
          in: path
          description: ID of pet to delete
          required: true
          schema:
            type: integer
            format: int64
      responses:
        '204':
          description: pet deleted
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
components:
  schemas:
    Pet:
      allOf:
        - $ref: '#/components/schemas/NewPet'
        - type: object
          required:
          - id
          properties:
            id:
              type: integer
              format: int64

    NewPet:
      type: object
      required:
        - name  
      properties:
        name:
          type: string
        tag:
          type: string    

    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string

Note that ref can also be used with other files, making modularity easy.

There's 3 ways we can do openapi:

• Zod -> ts types & validation & openapi spec

  • Very explicit

  • Requires a complex compile step & defining what zod types become spec where

  • Zod has automatic ts types without requiring compile step (since it's just explicit types)

  • Tool to convert: https://github.com/asteasolutions/zod-to-openapi

  • Example

    import { generateSchema } from '@anatine/zod-openapi';
    const aZodSchema = z.object({
      uid: z.string().nonempty(),
      firstName: z.string().min(2),
      lastName: z.string().optional(),
      email: z.string().email(),
      phoneNumber: z.string().min(10).optional(),
    })
    const myOpenApiSchema = generateSchema(aZodSchema);

    Generates:

    {
      "type": "object",
      "properties": {
        "uid": {
          "type": "string",
          "minLength": 1
        },
        "firstName": {
          "type": "string",
          "minLength": 2
        },
        "lastName": {
          "type": "string"
        },
        "email": {
          "type": "string",
          "format": "email"
        },
        "phoneNumber": {
          "type": "string",
          "minLength": 10
        }
      },
      "required": [
        "uid",
        "firstName",
        "email"
      ]
    }

• openapi spec -> ts types & validation
  • The most popular choice
  • Uses yaml files
  • Allows for referencing other files, so can be modularized like other solutions
  • Requires compile step to make ts types which can then be used
  • Tool to convert: https://www.npmjs.com/package/openapi-typescript
  • Validation is just middleware (https://www.npmjs.com/package/express-openapi-validator), and can validate input and output easily
• ts types -> openapi spec

  • Uses decorators on types

  • Integrates with jsdoc for text

  • Doesn't require having a separate file

  • Works best with service/controller/model structure

  • Tool to convert: https://github.com/lukeautry/tsoa

  • Example

    // src/users/usersController.ts
    import {
      Body,
      Controller,
      Get,
      Path,
      Post,
      Query,
      Route,
      SuccessResponse,
    } from "tsoa";
    import { User } from "./user";
    import { UsersService, UserCreationParams } from "./usersService";
    
    @Route("users")
    export class UsersController extends Controller {
      @Get("{userId}")
      public async getUser(
        @Path() userId: number,
        @Query() name?: string
      ): Promise<User> {
        return new UsersService().get(userId, name);
      }
    
      @SuccessResponse("201", "Created") // Custom success response
      @Post()
      public async createUser(
        @Body() requestBody: UserCreationParams
      ): Promise<void> {
        this.setStatus(201); // set return status 201
        new UsersService().create(requestBody);
        return;
      }
    }

There's a ton of other options, but I'd like to choose something that doesn't overcomplicate the API and avoids restructuring everything. Mainly, more of library than a framework.

[Timothy-Gonzalez]

Swagger makes this very nice:

Also, no need to use postman/insomniac/etc, you can just try requests:

[Timothy-Gonzalez]

Intro

I conducted an investigation into several alternatives by implementing them for a simple CRUD Users app.

Full Notes

The full thing is very verbose, so let me summarize.

Validation

First of all, Zod is awesome for validation and we should use it. There are other solutions like Typebox / json schema but they are much harder to use. Zod also integrated natively with ts types. (You can do type a = z.infer<typeof ZodSchema> to get the type from a schema. Here's an example:

export const UserSchema = z.object({
    id: UserIdSchema,
    userName: z
        .string()
        .min(1)
        .max(20)
        .regex(/^[a-z0-9]+$/, "Username must be alphanumeric"),
    displayName: z.string().min(1).max(50),
});
export type User = z.infer<typeof UserSchema>;

Documentation

I also determined we should use OpenAPI (a documentation specification based off of json schemas), since it is widely supported by a lot of tooling, namely Swagger (see above) which makes docs much nicer. Also, generation tooling around OpenAPI is great, so with just the spec you can get a client that lets you do client.getUser(id) for example, without writing any code.

The difficult part was figuring out HOW to integrate this together with our application (currently express).

There are two main options:

1. Define the OpenAPI spec and generate types/validation from it (OpenAPI -> Code)
2. Define routes/types and generate OpenAPI spec from it (Code -> OpenAPI)

The problem with the first option is the tooling for directly editing yaml/json aren't great, and modularizing the files requires a lot of ugly $ref-ing or a merge step which breaks the very little intellesense you get.

So, we're left with the second option. There's a lot of options in this subset, but it's more new. There's a lot of ways to do annotation through decorators, explicit definition, etc.

There's two main parts we need to convert into the spec: components and paths. Paths are routes like GET /user/ which need to define what they take in, and what they receive. Components are objects used in paths, like a User component.

Generating Components

For components, we can use a zod conversion library which adds metadata:

export const UserSchema = z.object({
    id: UserIdSchema,
    userName: z
        .string()
        .min(1)
        .max(20)
        .regex(/^[a-z0-9]+$/, "Username must be alphanumeric")
        .openapi({
            description: "The user's username, which must be alphanumeric",
            example: "username1",
        }),
    displayName: z.string().min(1).max(50).openapi({
        description: "The user's display name, with no limitations on characters allowed",
        example: "1 Full Display Name",
    }),
}).openapi({
    ref: "User",
    description: "A user"
});

Note that descriptions and examples can be added easily, and the ref is used to add the schema to the generated document.

Generating Paths

The last part we need to generate is paths, which there are many tools for.

One popular choice is tsoa which through decorators, you can define routes. The problem is this doesn't work very well with zod. This could be viable alternative, but the decorators are very verbose and not type-checked well.

One option is registering them directly in ts, next to route definitions:

Registry.registerPath({
    method: "post",
    path: "/users",
    summary: "Create a new user",
    request: {
        body: {
            content: {
                "application/json": {
                    schema: UserSchema,
                },
            },
        },
    },
    responses: {
        200: {
            description: "Successfully created user",
            content: {
                "application/json": {
                    schema: UserSchema,
                },
            },
        },
    },
});
usersRouter.post("/", validateRequestBody(UserSchema), (req, res) => {
    const newUser = createUser(req.body);

    res.status(200).json(newUser);
});

The better alternative is using express middleware:

usersRouter.post(
    "/",
    specification({
        summary: "Create a new user",
        body: UserSchema,
        responses: {
            200: {
                description: "Successfully created user",
                schema: UserSchema,
            },
        },
    }),
    (req, res) => {
        const newUser = createUser(req.body);

        res.status(200).json(newUser);
    },
);

However, this requires a lot of hacky code, and isn't as pretty behind the scenes. It feels like this really pushes express to the limit of what it was meant for.

Alternatively, using a REST library that support schema validation natively might be a better solution. Here's Fastify:

app.post(
    "/",
    {
        schema: {
            description: "Create a new user",
            body: UserSchema,
            response: {
                200: UserSchema.openapi({
                    description: "Successfully created user",
                }),
            },
        },
    },
    (req, _reply) => {
        return createUser(req.body);
    },
);

Fastify also supports proper typing, and allows you to return for successful requests without the whole .status .send:

app.get(
    "/:id",
    {
        schema: {
            description: "Get a user by id",
            params: z.object({
                id: UserIdSchema,
            }),
            response: {
                200: UserSchema.openapi({
                    description: "Successfully got user",
                }),
                404: UserNotFoundErrorSchema,
            },
        },
    },
    (req, reply) => {
        const user = getUser(req.params.id);

        if (!user) {
            reply.status(404).send(UserNotFoundError);
        }

        return user;
    },
);

Conclusion

In conclusion, OpenAPI and Zod are great tools we should use. For generation, we either stay with express and release the demons, or switch everything over to Fastify for questionable benefit. Whatever solution we choose, something in the middle won't really work. The whole point of this change is to have strict parity between the docs and code while changing as little as possible and keeping the codebase easy to work with.

[Timothy-Gonzalez]

Long term, I think the express middleware without the hacky part makes sense. To make this migration easy, there are 3 parts:

1. Migrate all docs to use specification middleware, creating schemas as needed
2. Merge zod schemas with mongoose schemas, long term might drop typegoose
3. Clean up error handling