Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Major refactoring to support more thorough conversion of dandischema pydantic models #1

Merged
merged 205 commits into from
Oct 15, 2024
Merged

Major refactoring to support more thorough conversion of dandischema pydantic models #1

merged 205 commits into from
Oct 15, 2024

Conversation

yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Jul 23, 2024 •
edited by candleindark
Loading

Done locally to replace

This PR improves the existing conversion from Pydantic models to LinkML the following areas:

  1. Improve enum conversion so that different slots within the EnumDefinition metamodel are populated more accurately
  2. Add more detailed slots population
  3. Add slot usage in class population
  4. Provide a more accurate interpretation of slots/attributes in a class, by eliminating misinterpretation of a class variable in a Pydantic model as a slot for example.
  5. Provide notes to inform issues encounter in the conversion
  6. Specify limitations of this conversion tool in README.md
  7. Provide more tests to improve coverage
  8. Reorganized code so that additional support can be added more easily
  9. Provide a solution in handling a Pydantic model that has multiple parents that are also Pydantic models.
  10. Organize this repo into a Python project using Hatch
  11. Enable CLI to be installed as part of the project installation.

@candleindark
Outline the basic structure of the LinkML generator
75e210a
@candleindark
Ensure no name collection in initiating LinkmlGenerator
76e052f
@candleindark
Change the signature of tools.ensure_unique_names
2a59078 
Allow classes to be input as positional argument makes
the function more portable
@candleindark
Add all enums to the schema
dce98dd
@candleindark
Define func to normalized whitespace
7974869 
This func is intended to be used for converting
a docstring to description in metadata in a
LinkML schema
@candleindark
Include description in EnumDefinition
1328950
@candleindark
Define interfaces to add slots and classes
23f1a05
@candleindark
Rearrange comments
eda50a7
@candleindark
Define internal map of models to their locally defined fields
dd4888c
@candleindark
Implement LinkmlGenerator._add_slots
8cfe73b 
A method that adds the slots construed from
the models in `self._models` to the schema
@candleindark
Provide a function to bucketize items
36f5cc6
@candleindark
Utilize bucketize
3f783b0
@candleindark
Add support of transforming values in bucketize
1d6a711
@candleindark
Adopt the new support in bucketize
1607e0c
@candleindark
Change slot generation to be handled by a class
f9025d2
@candleindark
Implement SlotGenerator.__init__
58135c1
@candleindark
Outline SlotGenerator.generate
22ef7b0
@candleindark
Specifies interfaces of Pydantic core schema handler functions in `Sl…
c515b05 
…otGenerator`
@candleindark
Reformat generate_linkml_from_aind.py with Black
bcfff9a
@candleindark
Remove unused imports
af93dce 
In generate_linkml_from_aind.py
@candleindark
Replace use of not and in with not in
0423fb8
@candleindark
Remove unused variable in generate_linkml_from_aind.main()
2f9b121
@candleindark
Correct type annotation in populate_enum and populate_basemodel
be16880
@candleindark
Move get_all_modules() to tools.py
674ad72
@candleindark
Use inner function to simplify the interface of get_all_modules()
ce5476e
@candleindark
Implement fetch_defs() and provided needed tests
afeecc9 
`fetch_defs()` is a function for fetching Python objects
from given modules that provide schema definitions
@candleindark
Corret import statement for CoreSchemaOrField
cb68232
@candleindark
Rename local variables
c17f55e
@candleindark
Correct retrieval of corresponding core_schema.ModelSchema of a Pyd…
268700b 
…antic model
@candleindark
Remove limitation of not supporting multi inheritance in Pydantic models
87c7ef0
@candleindark
Simply the key function for sorting slots by names
46ef009
@candleindark
Remove unneeded comment
134e685
@candleindark
Define SlotExtensionError
73f5815
@candleindark
Define get_non_empty_meta_slots()
b5155fd
@candleindark
Raise ValueError in SlotExtensionError creation
647ebda
@candleindark
Define get_slot_usage_entry
3af3e4e
@candleindark
RF: Introduce sort_dict_by_ikeys()
e736339
@candleindark
Introduce a dict for slot representation of new fields
3cab658
@candleindark
Add slot usage entries for new slots
cfc7261
@candleindark
Ensure slot usage entries are sorted
160a7d4
@candleindark
Delay sorting of slots
843d145
@candleindark
Rename variable
959b5e3
@candleindark
Improve docstring of SlotExtensionError
1e4a16b
@candleindark
Handling overriding fields in class generation
cf45afb
@candleindark
Set parent class of a class definition
4a6ab7c
@candleindark
Rename test_get_parent_model to test_get_parent_models
dd50a66
@candleindark
Make pydantic.BaseModel an invalid input to get_parent_models
c003aea 
`pydantic.BaseModel` doesn't really model anything. It is merely
a template to specify a model.
@candleindark
Specify mixins in generating a ClassDefinition
0810e8d
@candleindark
Specify mixins in generating a ClassDefinition
f839ad8
@candleindark
Add the interface for translating complex number specs
3ec2098 
Pydantic 2.9 adds support of validation of complex numbers.
To be compatible with Pydantic 2.9, We need this interface
defined.
@candleindark
Restrict pydantic to ~=2.9
f8e1491 
Pydantic 2.8 doesn't have "core_schema.ComplexSchema"
needed in the current code. Let's support Pydantic 2.8
only when needed
@candleindark
Update .gitignore to ignore __pycache__ folders
8bab6d1
@candleindark
Update README.md
640f9c6
@candleindark
Update project urls
27ec216 
Due to the transfer of the project to the DANDI
organization on github
@candleindark
fix: Make SlotGenerator._complex_schema() contingent on Pydantic ve…
9c33b17 
…rsion

With this contingency in place Pydantic no longer need to be
2.9 or greater
@candleindark
Copy link
Member

Merging this into main per @yarikoptic's request. The PR has already grown to the point that meaningful review is not practical.

@candleindark candleindark merged commit fdf4828 into main Oct 15, 2024
3 checks passed
@candleindark candleindark mentioned this pull request Oct 15, 2024
Closed
14 tasks
@candleindark candleindark deleted the enh branch October 15, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@candleindark candleindark candleindark left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yarikoptic @candleindark