feat: OpenAPI integration

[NexVeridian]

related #855

cargo check -F all_openapi
cargo nextest run --test-threads 1 -F all_openapi

docker run -d \
  -e POSTGRES_DB=postgres_test \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  --health-cmd="pg_isready" \
  --health-interval=10s \
  --health-timeout=5s \
  --health-retries=5 \
  postgres

export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/postgres_test"

cargo nextest run --all-features --workspace --exclude loco-gen --exclude loco --test-threads 1

TODO:

• config.yaml
  • maybe no enable: true since that's handled by the feature

    openapi:
      redoc:
        !Redoc
          url: /redoc
          spec_json_url: /redoc/openapi.json
          spec_yaml_url: /redoc/openapi.yaml
      scalar:
        !Scalar
          url: /scalar
          spec_json_url: /scalar/openapi.json
          spec_yaml_url: /scalar/openapi.yaml
      swagger:
        !Swagger
          url: /swagger-ui
          spec_json_url: /api-docs/openapi.json
          spec_yaml_url: /api-docs/openapi.yaml

  • finish .merge(Redoc::with_url("/redoc", api.clone()))
  • openapi.josn and openapi.yaml endpoints for all types
• split feature openapi into feature all_openapi swagger-ui redoc scalar
• rstest feature flagged cases
• SecurityAddon
  • put impl Modify for SecurityAddon somewhere, maybe with config
  • set the jwt token location
• tests
  • update src/tests_cfg/db.rs:86:1
  • src/tests_cfg/config.rs
  • config from file test_from_folder_openapi()
  • snapshots
• docs
• maybe auto fill / wrap utoipa::path if possible
• check that get in get(get_action_openapi) is still grabbed with routes!(get_action_openapi)
• fix AppContext - check that api_router.routes(method.with_state::<AppContext>(())) doesn't break the ctx with .layer
• cargo test is broken with JWT_LOCATION.get_or_init, nextest works correctly
• codegen
  • cargo loco generate controller --openapi
  • adding #[derive[utoipa::ToSchema)] if used in utoipa::path
  • routes! macro

cc @DenuxPlays

[DenuxPlays]

Looks like a good start.

I think the yaml also should contain a few other things:

• openapi and ui endpoints (I am not sure if this is want you mean in your description)
• whether it serves json, yaml or both (Perhaps this can be controlled by using features to exclude unnecessary dependencies)

[DenuxPlays]

Hey @NexVeridian

Can I help you finish this pr?

[NexVeridian]

@DenuxPlays yeah of course

[DenuxPlays]

How can I help?

Also just one Note to your pr description.
I wouldn't wrap the utoipa path macro.
Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

[NexVeridian]

Also just one Note to your pr description. I wouldn't wrap the utoipa path macro. Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

thanks!

this would be nice if possible:

impl Routes {
/// .add_openapi(routes!(get_action, post_action))
pub fn add_openapi(mut self, method: UtoipaMethodRouter<AppContext>) -> Self {}
}

any of the unchecked ones in the pr description would be great:
maybe cargo loco generate controller --openapi and docs first

[NexVeridian]

seems like this might be flaky in cargo test because the global state isn't reset between test runs
tests are good in cargo nextest and should be fine for users

[NexVeridian]

Just one small thing I noticed. Structs like these: https://github.com/NexVeridian/loco/blob/d72c46032a93a676004ade03266f76e6268b26bc/src/controller/views/pagination.rs would need to implement/derive the ToSchema trait right?

if it's used in the utoipa::path then yeah, I'm not sure if they would be, I haven't used those before

[DenuxPlays]

Just one small thing I noticed. Structs like these: https://github.com/NexVeridian/loco/blob/d72c46032a93a676004ade03266f76e6268b26bc/src/controller/views/pagination.rs would need to implement/derive the ToSchema trait right?

if it's used in the utoipa::path then yeah, I'm not sure if they would be, I haven't used those before

They would to provide Basic paginagtion metadata

[SorenEdwards]

@DenuxPlays @NexVeridian

I had to do something similar recently and If I understand correctly you would need to generate something like this. I had a problem where the generated openapi schema didn't see that X struct was a query parameter.

#[derive(Debug, Deserialize, Serialize,ToSchema,IntoParams)]
#[into_params(parameter_in=Query)]
pub struct PagerMeta {
    #[serde(rename(serialize = "page"))]
    pub page: u64,
    #[serde(rename(serialize = "page_size"))]
    pub page_size: u64,
    #[serde(rename(serialize = "total_pages"))]
    pub total_pages: u64,
}

[NexVeridian]

5f4c63f still has some errors that I don't know how to fix, one of you two should take a look maybe

[DenuxPlays]

I think the Solution would be something like cfg_of and we have to define the struct two times.

[SorenEdwards]

I think this would work, I tested it on your branch. The cfg_attr for the trait bounds is pretty ugly so I think @DenuxPlays has a point with using cfg_of and having two structs.

Pragnation.rs

#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::ToSchema)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct Pager<
#[cfg(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
))] T: utoipa::ToSchema,
#[cfg(not(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
),))] T,
> {
    #[serde(rename(serialize = "results"))]
    pub results: T,

    #[serde(rename(serialize = "pagination"))]
    pub info: PagerMeta,
}


#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::ToSchema)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct PagerMeta {
    #[serde(rename(serialize = "page"))]
    pub page: u64,
    #[serde(rename(serialize = "page_size"))]
    pub page_size: u64,
    #[serde(rename(serialize = "total_pages"))]
    pub total_pages: u64,
    #[serde(rename(serialize = "total_items"))]
    pub total_items: u64,
}

impl<
#[cfg(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
))] T: utoipa::ToSchema,
#[cfg(not(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
),))] T,
> Pager<T> {
    #[must_use]
    pub const fn new(results: T, meta: PagerMeta) -> Self {
        Self {
            results,
            info: meta,
        }
    }
}

pragnate/mod.rs

#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::IntoParams)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct PaginationQuery {
    #[serde(
        default = "default_page_size",
        rename = "page_size",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page_size: u64,
    #[serde(
        default = "default_page",
        rename = "page",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page: u64,
}

[DenuxPlays]

I had something like the following in mind.
Also this isn't tested I do not have time for this atm.

pagination.rs

use cfg_if::cfg_if;

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        #[derive(Debug, serde::Deserialize, serde::Serialize, utoipa::ToSchema)]
        pub struct Pager<T: utoipa::ToSchema> {
            #[serde(rename(serialize = "results"))]
            pub results: T,

            #[serde(rename(serialize = "pagination"))]
            pub info: PagerMeta,
        }
    } else {
        #[derive(Debug, serde::Deserialize, serde::Serialize)]
        pub struct Pager<T> {
            #[serde(rename(serialize = "results"))]
            pub results: T,

            #[serde(rename(serialize = "pagination"))]
            pub info: PagerMeta,
        }
    }
}

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        #[derive(Debug, serde::Deserialize, serde::Serialize, utoipa::ToSchema)]
        pub struct PagerMeta {
            #[serde(rename(serialize = "page"))]
            pub page: u64,
            #[serde(rename(serialize = "page_size"))]
            pub page_size: u64,
            #[serde(rename(serialize = "total_pages"))]
            pub total_pages: u64,
            #[serde(rename(serialize = "total_items"))]
            pub total_items: u64,
        }
    } else {
        #[derive(Debug, serde::Deserialize, serde::Serialize)]
        pub struct PagerMeta {
            #[serde(rename(serialize = "page"))]
            pub page: u64,
            #[serde(rename(serialize = "page_size"))]
            pub page_size: u64,
            #[serde(rename(serialize = "total_pages"))]
            pub total_pages: u64,
            #[serde(rename(serialize = "total_items"))]
            pub total_items: u64,
        }
    }
}

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        impl<T: utoipa::ToSchema> Pager<T> {
            #[must_use]
            pub const fn new(results: T, meta: PagerMeta) -> Self {
                Self {
                    results,
                    info: meta,
                }
            }
        }
    } else {
        impl<T> Pager<T> {
            #[must_use]
            pub const fn new(results: T, meta: PagerMeta) -> Self {
                Self {
                    results,
                    info: meta,
                }
            }
        }
    }
}

I think that @SorenEdwards implementation for the query is good but we should also add ToSchema to it.
It may be useful for some use cases.
So this is what I would do:

#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::IntoParams, utoipa::ToSchema)
)]
#[derive(Debug, serde::Deserialize, serde::Serialize)]
pub struct PaginationQuery {
    #[serde(
        default = "default_page_size",
        rename = "page_size",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page_size: u64,
    #[serde(
        default = "default_page",
        rename = "page",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page: u64,
}

[kaplanelad]

After reviewing this implementation again, I feel we should consider moving it into an initializer maintained in a separate repository. For example:

• loco-oauth2
• rhai-loco
• loco-extras

Could you outline the changes needed in Loco to allow managing OpenAPI as an initializer?

[DenuxPlays]

Note:

I don't think thats possible.
we need loco to implement certain traits for this to work.
examples are the above mentioned Pager and PagerMeta types

[kaplanelad]

Note:

I don't think thats possible. we need loco to implement certain traits for this to work. examples are the above mentioned Pager and PagerMeta types

Everything is possible. When initializing is public create, you need to load Loco (see the other implementation), which allows us to make the dependent struct public.
I want to count the changes that we need to make in the framework to make it happen

[SorenEdwards]

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

[DenuxPlays]

Note:

I don't think thats possible. we need loco to implement certain traits for this to work. examples are the above mentioned Pager and PagerMeta types

Everything is possible. When initializing is public create, you need to load Loco (see the other implementation), which allows us to make the dependent struct public.

I want to count the changes that we need to make in the framework to make it happen

I am not sure what you mean.
AFAIK traits can only be implemented on internal, your own types or if you created the trait.

So unless you copy the structs it isnt possible right?

[DenuxPlays]

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

Yeah I also use it as an initializer.
But I don't think that Code Generation is my Main aspect.
Currently I use utoipauto to collect all the paths, schemas etc. But this would actually use the utoipa-Axum Router which is far cleaner then using utoipauto.
Also you do not have to copy types like Pager.

It is just a Smoother experience.

Just a small note you should Never Serve entities over your api.
Just use DTOs/Views for this.
Ofcourse I do Not know your project and it may be not the best Solution for your I just wanted to mention it.

[kaplanelad]

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

Yes, it can be great. can you also share how you implement it with an example repo?

[DenuxPlays]

You can take look here:
https://github.com/financrr/financrr-app/blob/backend/feature/sync-bank-data/backend/src/initializers/openapi.rs

[kaplanelad]

After reviewing this implementation again, I feel we should consider moving it into an initializer maintained in a separate repository. For example:

• loco-oauth2
• rhai-loco
• loco-extras

Could you outline the changes needed in Loco to allow managing OpenAPI as an initializer?

@NexVeridian ?

[NexVeridian]

@kaplanelad

There are some issues with some of the options, like creating code generation in the future might be harder, here are some options for converting this pr to a initializer:

option 1 - probably not possible

Get the original function from a axum::routing::MethodRouter or use axum::routing::MethodRouter in utopia

option 2

have the user manually add the data to the initializer, similar to #855

# controllers::auth
pub fn api_routes() -> OpenApiRouter<AppContext> {
    OpenApiRouter::new().routes(routes!(register, verify, login, forgot, reset))
}

then pass controllers::auth::api_routes and others to the initializer

 let (_, api) = OpenApiRouter::with_openapi(ApiDoc::openapi())
            .merge(controllers::auth::api_routes())
            .merge(controllers::responses::api_routes())
            .split_for_parts();

option 3

Merge these two into loco:

• controller/routes.rs - LocoMethodRouter
  • so the user would call Routes::new().add("/_ping", routes!(ping));

• loco/src/controller/app_routes.rs

  Lines 218 to 237 in 5f4c63f

  	let mut api_router: OpenApiRouter<AppContext> =
  	OpenApiRouter::with_openapi(H::inital_openapi_spec(&ctx));
  	for router in self.collect() {
  	tracing::info!("{}", router.to_string());
  	match router.method {
  	LocoMethodRouter::Axum(method) => {
  	app = app.route(&router.uri, method);
  	}
  	#[cfg(any(
  	feature = "openapi_swagger",
  	feature = "openapi_redoc",
  	feature = "openapi_scalar"
  	))]
  	LocoMethodRouter::Utoipa(method) => {
  	app = app.route(&router.uri, method.2.clone());
  	api_router = api_router.routes(method.with_state(ctx.clone()));
  	}
  	}
  	}

  • then here you would pass api_router or vec!method.with_state(ctx.clone()) to the initializer

option 4

This comment is also a good option but has these drawbacks, as stated in the comment:

Yeah I also use it as an initializer.
But I don't think that Code Generation is my Main aspect.
Currently I use utoipauto to collect all the paths, schemas etc. But this would actually use the utoipa-Axum Router which is far cleaner then using utoipauto.

[kaplanelad]

All the options look legitimate to me when we move them to the initializer.
I can create a project under Loco, and we can start working on it together.

WDYT?

[DenuxPlays]

What do you mean?

We cannot convert this feature to an initializer at least not without dropping 50% of the features.

[kaplanelad]

Could you please clarify which features are going to be supported and which ones won’t,and why?

I’d prefer to support this feature as an initializer (that is supported by Loco teams) and keep them separate from the base Loco codebase.

[DenuxPlays]

1. automatic Schema collection (includign routes, responses etc.)
   Because internally Loco would still use the axum router instead of the utoipa-axum router.

2. Pagination etc. cannot work with utoipa as they do not derive the Schema, Response traits etc.
   There is a work around that you just copy the code but tbh. copy code is always bad

There are probably some other things I forgot but the integration wouldn't feel like an integration

[kaplanelad]

1. automatic Schema collection (includign routes, responses etc.)
   Because internally Loco would still use the axum router instead of the utoipa-axum router.

Why not make the internal Loco route responses public so they can be accessed externally? This way, the initializer can retrieve the app context, which provides access to everything required by OpenAPI.

2. Pagination etc. cannot work with utoipa as they do not derive the Schema, Response traits etc.
   There is a work around that you just copy the code but tbh. copy code is always bad

There are probably some other things I forgot but the integration wouldn't feel like an integration

Same for here, we can expose it

[DenuxPlays]

I am not sure what you mean.

How can you "expose" something that needs to replaced when you want to use it with utoipa?

Also the struct is already exposed but the derive(Schema) is missing.
How is this fixable?

[NexVeridian]

keep them separate from the base Loco codebase

option 2:

this option removes automatic schema collection
and is the easiest to implement
and has almost no code that needs to merged into loco, just src/controller/format.rs and some derive(utoipa::ToSchema)

option 3:

this option has automatic schema collection
but is harder to implement
and a lot of the code in this pr will need to be merged

the only things that can be extracted is: the config, the tests, and this section app_routes.rs#L239-L290
and the LocoMethodRouter and app_routes.rs would need to stay

option 1:

Why not make the internal Loco route responses public so they can be accessed externally? This way, the initializer can retrieve the app context, which provides access to everything required by OpenAPI.

this is talking about option 1, which is probably not possible or I can't figure it out, maybe if you do something that looks bad like this:Routes::new().add("/_ping", "get", ping);