Replies: 5 comments 23 replies
-
|
Hi, dealing with large description documents? But 56MB is … interesting. First - convert to json, test loading speed with orjson, pycsimdjson, ijson Second - as you already do, limit models to the models you need. class SchemaSelector(Init):
"""
remove the schemas we do not need models for
"""
def __init__(self, *schemas):
super().__init__()
self.schemas = schemas
def schema(self, ctx: "Init.Context") -> "Init.Context":
ctx.schema = {k: ctx.schema[k] for k in (set(self.schemas) & set(ctx.schema.keys()))}
return ctxThe logic will resolve required/related/linked models - as your own logic does. remove pathes you do not need class RemovePaths(Document):
def parsed(self, ctx: "Document.Context") -> "Document.Context":
"""
emtpy the paths - not needed
"""
ctx.document["paths"] = {}
return ctxLast, do it "once" - once you are satisfied with the result, use cache_store/cache_load. You can profile easily. Currently does not load … required, nullable pathparameter which does not exist - I consider this invalid. Now it's taking me about 6 GB RAM and I'll submit this before my machine dies oom … . _init_schema_types() creates the model classes from the json schemas - time which needs to be spent. |
Beta Was this translation helpful? Give feedback.
-
|
Would have to run on a machine more capable than my notebook. Forgot to mention .clone() and could have linked cache_* |
Beta Was this translation helpful? Give feedback.
-
Interesting. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, the MS Graph API is pretty extensive and not perfect. In this situation, I'm currently using Something I should be using more is the Plugin functionality you've already built out. I'll probably dive into that tomorrow for generating and cleaning up the definitions. I also have some ideas for possible performance improvements with the Thank you, as always, for your insights! |
Beta Was this translation helpful? Give feedback.
-
model_rebuild is pydantic - without involving the pydantic crew this has to be considered static. I pushed mypy branch for you to work with. The custom schema selection could/has to be be improved to match yours - work on a PathItem basis and include only the required schemas for the operations chosen. I think about skipping the components.schemas lookup in _init_schema_types_collect in certain conditions - maybe the self._root.paths got changed (by len() or id()) _init_operationindex, we'll see … Great to see you are putting the library to good use. |
Beta Was this translation helpful? Give feedback.
-
|
Cloned the mypy branch, and I'm running in to this. Not an issue with the branch, but with pydantic.
|
Beta Was this translation helpful? Give feedback.
-
|
This works: |
Beta Was this translation helpful? Give feedback.
-
|
The ref to pydantic@github is a relict from the v2 migration. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a functional purpose for using a ChainMap in this method? aiopenapi3/aiopenapi3/openapi.py Lines 399 to 459 in 9947b29 I did some testing and iterating without recursion, using just For my tests, the |
Beta Was this translation helpful? Give feedback.
-
|
simdjson is fast but looses when all fields get accessed to orjson which is faster than ujson. As startup reduction is way easier and has the benefit to allow removing all not required data from memory … I gave it a shot. This is an easy way to test it … using tests/reduced_test.py with class PetStoreReduced(Reduced):
def __init__(self):
super().__init__({"/user/{username}":None})the ms-graph api reduced to one operation and tracemalloc is quite slow, without … It's most likely the same as you do to reduce description documents but embedded. |
Beta Was this translation helpful? Give feedback.
-
|
I refactored my reduction workflow to use The resulting JSON is over 5MB (without any formatting). Certainly more manageable than 56MB, but front-loading all the OpenAPI schemas as Pydantic models still takes a like 6-8min. I have to jump into some other projects this week, but I'll look into the lazy-loading idea we talked about next week. |
Beta Was this translation helpful? Give feedback.
-
|
For your example with 2 paths on msgraph - 5s total. |
Beta Was this translation helpful? Give feedback.
-
|
Had a chance this afternoon to test the That particular reference uses a discriminator that contains, presumably, every Seems to me the best course of action would be to piggyback off the Reduced plugin to strip these unnecessary elements out. |
Beta Was this translation helpful? Give feedback.
-
|
I reproduced this in a unit test and fixed it - missing dependency tracking for discriminator mapping schemas. |
Beta Was this translation helpful? Give feedback.
-
|
Curious of your thoughts on my changes here: ggpwnkthx@645f1af
Sub 4 second but I'm culling things from the |
Beta Was this translation helpful? Give feedback.
-
|
The ref tracking is impressive - I like it. The list comprehensions aren't easy to read, black obfuscates them even more, but I see the speed benefit. The response type correction - I'd expect removing components.schemas."microsoft.graph.entity".discriminator would do as well. def parsed(self, ctx: "Document.Context") -> "Document.Context":
super().parsed(ctx)
del ctx.document["components"]["schemas"]["microsoft.graph.entity"]["discriminator"]
return ctxThe entity object can not be instantiated anyway, it's discriminator is shared with it's discriminated unions. 3.3s is impressive |
Beta Was this translation helpful? Give feedback.
-
|
One |
Beta Was this translation helpful? Give feedback.
-
|
Is If so, that plugin method doesn't ever get called if the OpenAPI object is handled via the class constructor - only if a I presume, at least in the vast majority of cases, the document has to be loaded from somewhere. So, it'd be rare that an OpenAPI object ever needs to be initialized directly by the constructor, and if it is needed, it's reasonable to require that the definition is in its final form. Which is to say, I think the answer to my question is most likely, yes - |
Beta Was this translation helpful? Give feedback.
-
|
I consider parsed to be the right place - I know it does not play nice with OpenAPI.init. |
Beta Was this translation helpful? Give feedback.
-
|
Just pushed to my fork an update that handles things with class inheritance. I'm not 100% on their names, though. |
Beta Was this translation helpful? Give feedback.
-
|
cherry picked - master...reduced Looking good to you? |
Beta Was this translation helpful? Give feedback.
-
|
I put two comments in there on the documentation. Otherwise, I think it's great! |
Beta Was this translation helpful? Give feedback.
-
|
merged reduced in #162 thanks |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your patience, cooperation, and insights! Is there anything needed from my fork? If not, I'll delete it and close out this discussion. |
Beta Was this translation helpful? Give feedback.
-
|
It's been a pleasure. I'll reach out if I feel the need to cull multi file large description documents. |
Beta Was this translation helpful? Give feedback.
-
I'm trying to work with some rather large OpenAPI files (specifically this one Microsoft Graph OpenAPI). Since the majority of that file is unnecessary to load at runtime (especially when I may only need one endpoint for a given process/instance), I wrote some code that essentially indexes the components and paths into a zip file as JSON data, as well as some code to generate an OpenAPI file based the desired paths and any
$refs.This works well as it only takes about a minute to generate the zip file and less than a second to compile only the necessary assets. That said, sometimes I'm still left with definitions that can be 100k-200k lines long (in YAML format). The method that seems to take the longest to process is
_init_schema_typesaiopenapi3/aiopenapi3/openapi.py
Line 467 in 0a0789c
I was going to dive into this and see what I could improve on, but figured I'd touch base to see if you had any thoughts.
Beta Was this translation helpful? Give feedback.
All reactions