Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⬆️ Upgrade models-library (pydantic v2) #6333

Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
6c6bebc
remove version constraint
giancarloromeo Sep 9, 2024
533bdd1
upgrade requirements
giancarloromeo Sep 9, 2024
c1bbc52
run bump-pydantic
giancarloromeo Sep 9, 2024
76f0fc6
fix bump-pydantic issues
giancarloromeo Sep 9, 2024
55b72c7
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
df01a19
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
41b8797
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
7c61b0e
add pydantic-settings dependency
giancarloromeo Sep 10, 2024
e4c30e0
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
f25f246
add pydantic-extra-types dependency
giancarloromeo Sep 10, 2024
ba507d7
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
f2c72fe
fix bump-pydantic issues
giancarloromeo Sep 10, 2024
4f2d46d
fix field names
giancarloromeo Sep 10, 2024
e62831c
fix schemas
giancarloromeo Sep 10, 2024
4a2d1c3
fix model_config refs
giancarloromeo Sep 10, 2024
66edc00
fix basic types
giancarloromeo Sep 11, 2024
b4b8c87
fix rabbitmq basic types
giancarloromeo Sep 11, 2024
00b2f6d
fix double config field
giancarloromeo Sep 11, 2024
36a15e1
fix projects issues
giancarloromeo Sep 11, 2024
ffca801
use pydantic annotated types
giancarloromeo Sep 11, 2024
5812571
continue refactoring
giancarloromeo Sep 11, 2024
08e7b50
fix model validator
giancarloromeo Sep 11, 2024
e2895bc
upgrade fastapi encoders
giancarloromeo Sep 11, 2024
ac68f4b
fix root models
giancarloromeo Sep 11, 2024
c5d2531
fix mapping
giancarloromeo Sep 11, 2024
f7356de
fix json serialization
giancarloromeo Sep 11, 2024
2235ec6
fix config dicts merge
giancarloromeo Sep 11, 2024
e89ea76
continue upgrading
giancarloromeo Sep 11, 2024
689bf92
continue upgrading
giancarloromeo Sep 11, 2024
7fb1191
continue upgrading
giancarloromeo Sep 12, 2024
78a1571
continue upgrading
giancarloromeo Sep 12, 2024
6361439
continue upgrading
giancarloromeo Sep 12, 2024
2962805
fix emails
giancarloromeo Sep 12, 2024
5afa03f
fix errors
giancarloromeo Sep 12, 2024
53db14a
fix more
giancarloromeo Sep 12, 2024
7db6bc9
continue fixing
giancarloromeo Sep 12, 2024
40913c4
fix node config
giancarloromeo Sep 12, 2024
c1ecea5
fix default validator
giancarloromeo Sep 13, 2024
bb24bee
fix NodeId type
giancarloromeo Sep 13, 2024
e65670d
update node
giancarloromeo Sep 13, 2024
23d2147
fix union mode
giancarloromeo Sep 13, 2024
31695ce
fix constrained str
giancarloromeo Sep 13, 2024
27217c6
fix project locked
giancarloromeo Sep 13, 2024
9b32fa2
fix model configs
giancarloromeo Sep 13, 2024
5e92cc1
fix httpurl
giancarloromeo Sep 13, 2024
debd71c
fix service settings labels
giancarloromeo Sep 13, 2024
901d6ee
fix settings labels validators
giancarloromeo Sep 13, 2024
bb48cd5
fix osparc variable identifier model
giancarloromeo Sep 15, 2024
4c66094
fix HttpUrl type
giancarloromeo Sep 15, 2024
0baf9ab
continue fixing
giancarloromeo Sep 15, 2024
e46c054
fix tests
giancarloromeo Sep 15, 2024
4b1f9fe
fix models equality check
giancarloromeo Sep 16, 2024
2941dea
fix field property
giancarloromeo Sep 16, 2024
a1be63c
fix mypy
giancarloromeo Sep 16, 2024
be4e077
fix mypy
giancarloromeo Sep 16, 2024
7e134cd
add validate default to thumbnail field
giancarloromeo Sep 16, 2024
3ba5029
fix validation issues
giancarloromeo Sep 16, 2024
b328087
fix model validator mode
giancarloromeo Sep 16, 2024
598b9dd
Update packages/models-library/src/models_library/api_schemas_catalog…
giancarloromeo Sep 16, 2024
cecf668
fix thumbnail type
giancarloromeo Sep 16, 2024
a6a86e0
remove empty model config
giancarloromeo Sep 16, 2024
0abee29
update format
giancarloromeo Sep 16, 2024
c4104c7
add typealias
giancarloromeo Sep 17, 2024
bafab91
revert bump-pydantic changes
giancarloromeo Sep 17, 2024
ea624d1
revert change
giancarloromeo Sep 17, 2024
88ec517
fix union mode
giancarloromeo Sep 17, 2024
b62770d
fix default value
giancarloromeo Sep 17, 2024
96a93ac
fix url types
giancarloromeo Sep 17, 2024
cbf481e
fix format
giancarloromeo Sep 17, 2024
2c18b4f
fix format
giancarloromeo Sep 17, 2024
c97244f
fix S3BucketName validation
giancarloromeo Sep 18, 2024
c049903
remove unnecessary todos
giancarloromeo Sep 18, 2024
65ced45
fix codesmell
giancarloromeo Sep 18, 2024
223bbcc
fix codesmell
giancarloromeo Sep 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/models-library/requirements/_base.in
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ arrow
jsonschema
orjson
pydantic[email]
pydantic-settings
pydantic-extra-types
18 changes: 16 additions & 2 deletions packages/models-library/requirements/_base.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
annotated-types==0.7.0
# via pydantic
arrow==1.3.0
# via -r requirements/_base.in
attrs==24.2.0
Expand All @@ -18,12 +20,22 @@ orjson==3.10.7
# via
# -c requirements/../../../requirements/constraints.txt
# -r requirements/_base.in
pydantic==1.10.17
pydantic==2.9.1
# via
# -c requirements/../../../requirements/constraints.txt
# -r requirements/_base.in
# pydantic-extra-types
# pydantic-settings
pydantic-core==2.23.3
# via pydantic
pydantic-extra-types==2.9.0
# via -r requirements/_base.in
pydantic-settings==2.4.0
# via -r requirements/_base.in
python-dateutil==2.9.0.post0
# via arrow
python-dotenv==1.0.1
# via pydantic-settings
referencing==0.35.1
# via
# jsonschema
Expand All @@ -37,4 +49,6 @@ six==1.16.0
types-python-dateutil==2.9.0.20240821
# via arrow
typing-extensions==4.12.2
# via pydantic
# via
# pydantic
# pydantic-core
10 changes: 3 additions & 7 deletions packages/models-library/requirements/_test.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@ coverage==7.6.1
# via
# -r requirements/_test.in
# pytest-cov
exceptiongroup==1.2.2
# via pytest
faker==27.0.0
# via -r requirements/_test.in
flexcache==0.3
Expand Down Expand Up @@ -68,7 +66,9 @@ python-dateutil==2.9.0.post0
# -c requirements/_base.txt
# faker
python-dotenv==1.0.1
# via -r requirements/_test.in
# via
# -c requirements/_base.txt
# -r requirements/_test.in
pyyaml==6.0.2
# via
# -c requirements/../../../requirements/constraints.txt
Expand All @@ -87,10 +87,6 @@ six==1.16.0
# python-dateutil
termcolor==2.4.0
# via pytest-sugar
tomli==2.0.1
# via
# coverage
# pytest
types-jsonschema==4.23.0.20240813
# via -r requirements/_test.in
types-pyyaml==6.0.12.20240808
Expand Down
10 changes: 0 additions & 10 deletions packages/models-library/requirements/_tools.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,6 @@ setuptools==73.0.1
# via pip-tools
shellingham==1.5.4
# via typer
tomli==2.0.1
# via
# -c requirements/_test.txt
# black
# build
# mypy
# pip-tools
# pylint
tomlkit==0.13.2
# via pylint
typer==0.12.4
Expand All @@ -94,8 +86,6 @@ typing-extensions==4.12.2
# via
# -c requirements/_base.txt
# -c requirements/_test.txt
# astroid
# black
# mypy
# typer
virtualenv==20.26.3
Expand Down
12 changes: 5 additions & 7 deletions packages/models-library/scripts/validate-pg-projects.py
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,10 @@

import typer
from models_library.projects import ProjectAtDB
from pydantic import Json, ValidationError, validator
from pydantic.main import Extra
from pydantic import ConfigDict, Json, ValidationError, field_validator


class ProjectFromCsv(ProjectAtDB):
class Config(ProjectAtDB.Config):
extra = Extra.forbid

# TODO: missing in ProjectAtDB
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please check if this comment has any relevance and just drop it if not?


access_rights: Json
Expand All @@ -22,17 +18,19 @@ class Config(ProjectAtDB.Config):

hidden: bool

model_config = ConfigDict(extra="forbid")

# NOTE: validators introduced to parse CSV

@validator("published", "hidden", pre=True, check_fields=False)
@field_validator("published", "hidden", mode="before", check_fields=False)
@classmethod
def empty_str_as_false(cls, v):
# See booleans for >v1.0 https://pydantic-docs.helpmanual.io/usage/types/#booleans
if isinstance(v, str) and v == "":
return False
return v

@validator("workbench", pre=True, check_fields=False)
@field_validator("workbench", mode="before", check_fields=False)
@classmethod
def jsonstr_to_dict(cls, v):
if isinstance(v, str):
Expand Down
5 changes: 2 additions & 3 deletions packages/models-library/src/models_library/access_rights.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
from pydantic import BaseModel, Extra, Field
from pydantic import BaseModel, ConfigDict, Field


class AccessRights(BaseModel):
read: bool = Field(..., description="has read access")
write: bool = Field(..., description="has write access")
delete: bool = Field(..., description="has deletion rights")

class Config:
extra = Extra.forbid
model_config = ConfigDict(extra="forbid")
11 changes: 4 additions & 7 deletions packages/models-library/src/models_library/aiodocker_api.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from pydantic import Field, validator
from pydantic import ConfigDict, Field, field_validator

from .generated_models.docker_rest_api import (
ContainerSpec,
Expand All @@ -16,7 +16,7 @@ class AioDockerContainerSpec(ContainerSpec):
description="aiodocker expects here a dictionary and re-convert it back internally`.\n",
)

@validator("Env", pre=True)
@field_validator("Env", mode="before")
@classmethod
def convert_list_to_dict(cls, v):
if v is not None and isinstance(v, list):
Expand All @@ -37,8 +37,7 @@ class AioDockerResources1(Resources1):
None, description="Define resources reservation.", alias="Reservations"
)

class Config(Resources1.Config): # type: ignore
allow_population_by_field_name = True
model_config = ConfigDict(populate_by_name=True)


class AioDockerTaskSpec(TaskSpec):
Expand All @@ -55,6 +54,4 @@ class AioDockerTaskSpec(TaskSpec):
class AioDockerServiceSpec(ServiceSpec):
TaskTemplate: AioDockerTaskSpec | None = None

class Config(ServiceSpec.Config): # type: ignore
alias_generator = camel_to_snake
allow_population_by_field_name = True
model_config = ConfigDict(populate_by_name=True, alias_generator=camel_to_snake)
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
from typing import Any, ClassVar

from pydantic import BaseModel, Field
from pydantic import BaseModel, ConfigDict, Field

from ..basic_types import VersionStr

Expand All @@ -12,11 +10,12 @@ class BaseMeta(BaseModel):
default=None, description="Maps every route's path tag with a released version"
)

class Config:
schema_extra: ClassVar[dict[str, Any]] = {
model_config = ConfigDict(
json_schema_extra={
"example": {
"name": "simcore_service_foo",
"version": "2.4.45",
"released": {"v1": "1.3.4", "v2": "2.4.45"},
}
}
)
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from pydantic import BaseModel, Field, SecretStr
from pydantic import BaseModel, ConfigDict, Field, SecretStr


class ApiKey(BaseModel):
Expand All @@ -15,5 +15,4 @@ class ApiKeyInDB(BaseModel):
user_id: int
product_name: str

class Config:
orm_mode = True
model_config = ConfigDict(from_attributes=True)
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
from datetime import datetime
from typing import Any, ClassVar, TypeAlias
from typing import Any, TypeAlias

from models_library.rpc_pagination import PageRpc
from pydantic import BaseModel, Extra, Field, HttpUrl, NonNegativeInt
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, NonNegativeInt

from ..boot_options import BootOptions
from ..emails import LowerCaseEmailStr
Expand All @@ -23,23 +23,23 @@


class ServiceUpdate(ServiceMetaDataEditable, ServiceAccessRights):
class Config:
schema_extra: ClassVar[dict[str, Any]] = {
model_config = ConfigDict(
json_schema_extra={
"example": {
# ServiceAccessRights
"accessRights": {
1: {
"execute_access": False,
"write_access": False,
},
}, # type: ignore[dict-item]
Copy link
Contributor Author

@giancarloromeo giancarloromeo Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In pydantic v2:

json_schema_extra: JsonSchemaExtraCallable | JsonDict | None

and JsonDict is defined as:

JsonDict: TypeAlias = Dict[str, JsonValue]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here we use an int instead of a str. Is that invalid JSON?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that example has to be stored as json.dumps?

2: {
"execute_access": True,
"write_access": True,
},
}, # type: ignore[dict-item]
44: {
"execute_access": False,
"write_access": False,
},
}, # type: ignore[dict-item]
},
# ServiceMetaData = ServiceCommonData +
"name": "My Human Readable Service Name",
Expand Down Expand Up @@ -72,6 +72,7 @@ class Config:
},
}
}
)


_EXAMPLE_FILEPICKER: dict[str, Any] = {
Expand Down Expand Up @@ -206,12 +207,11 @@ class ServiceGet(
): # pylint: disable=too-many-ancestors
owner: LowerCaseEmailStr | None

class Config:
allow_population_by_field_name = True
extra = Extra.ignore
schema_extra: ClassVar[dict[str, Any]] = {
"examples": [_EXAMPLE_FILEPICKER, _EXAMPLE_SLEEPER]
}
model_config = ConfigDict(
extra="ignore",
populate_by_name=True,
json_schema_extra={"examples": [_EXAMPLE_FILEPICKER, _EXAMPLE_SLEEPER]},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting that here it just works

)


class ServiceGetV2(BaseModel):
Expand All @@ -229,7 +229,7 @@ class ServiceGetV2(BaseModel):
service_type: ServiceType = Field(default=..., alias="type")

contact: LowerCaseEmailStr | None
authors: list[Author] = Field(..., min_items=1)
authors: list[Author] = Field(..., min_length=1)
giancarloromeo marked this conversation as resolved.
Show resolved Hide resolved
owner: LowerCaseEmailStr | None

inputs: ServiceInputsDict
Expand All @@ -249,11 +249,11 @@ class ServiceGetV2(BaseModel):
" It includes current release.",
)

class Config:
extra = Extra.forbid
alias_generator = snake_to_camel
allow_population_by_field_name = True
schema_extra: ClassVar[dict[str, Any]] = {
model_config = ConfigDict(
extra="forbid",
populate_by_name=True,
alias_generator=snake_to_camel,
json_schema_extra={
"examples": [
{
**_EXAMPLE_SLEEPER, # v2.2.1 (latest)
Expand Down Expand Up @@ -304,7 +304,8 @@ class Config:
],
},
]
}
},
)


PageRpcServicesGetV2: TypeAlias = PageRpc[
Expand All @@ -330,12 +331,11 @@ class ServiceUpdateV2(BaseModel):

access_rights: dict[GroupID, ServiceGroupAccessRightsV2] | None = None

class Config:
extra = Extra.forbid
alias_generator = snake_to_camel
allow_population_by_field_name = True
model_config = ConfigDict(
extra="forbid", populate_by_name=True, alias_generator=snake_to_camel
giancarloromeo marked this conversation as resolved.
Show resolved Hide resolved
)


assert set(ServiceUpdateV2.__fields__.keys()) - set( # nosec
ServiceGetV2.__fields__.keys()
assert set(ServiceUpdateV2.model_fields.keys()) - set( # nosec
ServiceGetV2.model_fields.keys()
) == {"deprecated"}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Any, ClassVar, Literal
from typing import Any, Literal

from pydantic import BaseModel, Field
from pydantic import BaseModel, ConfigDict, Field

from ..basic_regex import PUBLIC_VARIABLE_NAME_RE
from ..services import ServiceInput, ServiceOutput
Expand All @@ -17,7 +17,7 @@ class ServicePortGet(BaseModel):
key: str = Field(
...,
description="port identifier name",
regex=PUBLIC_VARIABLE_NAME_RE,
pattern=PUBLIC_VARIABLE_NAME_RE,
title="Key name",
)
kind: PortKindStr
Expand All @@ -26,9 +26,8 @@ class ServicePortGet(BaseModel):
None,
description="jsonschema for the port's value. SEE https://json-schema.org/understanding-json-schema/",
)

class Config:
schema_extra: ClassVar[dict[str, Any]] = {
model_config = ConfigDict(
json_schema_extra={
"example": {
"key": "input_1",
"kind": "input",
Expand All @@ -41,6 +40,7 @@ class Config:
},
}
}
)

@classmethod
def from_service_io(
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from pydantic import BaseModel, Field
from pydantic import BaseModel, ConfigDict, Field

from ..generated_models.docker_rest_api import ServiceSpec as DockerServiceSpec

Expand All @@ -12,9 +12,7 @@ class ServiceSpecifications(BaseModel):
default=None,
description="schedule-time specifications specifications for the service (follows Docker Service creation API (specifically only the Resources part), see https://docs.docker.com/engine/api/v1.41/#tag/Service/operation/ServiceCreate",
)

class Config:
pass
model_config = ConfigDict()

giancarloromeo marked this conversation as resolved.
Show resolved Hide resolved

class ServiceSpecificationsGet(ServiceSpecifications):
Expand Down
Loading
Loading